2020-03-11T22:19:25.245 INFO:root:teuthology version: 1.0.0-1def34b 2020-03-11T22:19:25.245 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.pTUUT3.tmp -- --verbose --lock --description rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} --name dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi --block --owner scheduled_dzafman@bionic-laptop --archive /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848251 2020-03-11T22:19:25.266 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:19:25.329 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848251 branch: wip-zafman-octo-diag description: rados/standalone/{supported-random-distro$/{centos_8.yaml} workloads/osd.yaml} email: dzafman@redhat.com first_in_suite: false job_id: '4848251' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.1' overrides: admin_socket: branch: wip-zafman-octo-diag ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-zafman-octo-diag sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 owner: scheduled_dzafman@bionic-laptop priority: 20 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 suite: rados suite_branch: wip-zafman-octo-diag suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 tasks: - install: null - workunit: basedir: qa/standalone clients: all: - osd teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.491 2020-03-11T22:19:25.329 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa; will attempt to use it 2020-03-11T22:19:25.330 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-zafman-octo-diag/qa/tasks 2020-03-11T22:19:25.330 INFO:teuthology.run_tasks:Running task internal.check_packages... 2020-03-11T22:19:25.332 INFO:teuthology.task.internal:Checking packages... 2020-03-11T22:19:25.346 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '4acc1121e47039a3cdf2978ae458c10d153aa141' 2020-03-11T22:19:25.346 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2020-03-11T22:19:25.346 INFO:teuthology.packaging:ref: None 2020-03-11T22:19:25.346 INFO:teuthology.packaging:tag: None 2020-03-11T22:19:25.346 INFO:teuthology.packaging:branch: wip-zafman-octo-diag 2020-03-11T22:19:25.346 INFO:teuthology.packaging:sha1: 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:19:25.347 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-zafman-octo-diag 2020-03-11T22:19:25.568 INFO:teuthology.task.internal:Found packages for ceph version 15.1.0-1941.g4acc112 2020-03-11T22:19:25.568 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2020-03-11T22:19:25.577 INFO:teuthology.task.internal:no buildpackages task found 2020-03-11T22:19:25.578 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2020-03-11T22:19:25.730 INFO:teuthology.task.internal.lock_machines:Locking machines... 2020-03-11T22:19:25.730 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:19:25.782 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:19:35.800 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:19:45.821 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:19:55.833 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:20:05.851 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:20:15.867 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:20:25.884 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:20:35.895 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:20:45.912 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 1, have 2)... 2020-03-11T22:20:55.927 DEBUG:teuthology.lock.ops:lock_many request: {'count': 1, 'locked_by': 'scheduled_dzafman@bionic-laptop', 'description': '/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848251', 'machine_type': 'smithi'} 2020-03-11T22:20:55.944 DEBUG:teuthology.lock.ops:locked ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:20:55.972 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:20:56.182 INFO:teuthology.provision.fog.smithi155:Scheduling deploy of centos 8.1 2020-03-11T22:20:56.566 INFO:teuthology.orchestra.console:Power off smithi155 2020-03-11T22:20:56.566 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2020-03-11T22:20:56.589 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:00.614 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:04.726 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:08.838 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:09.047 INFO:teuthology.orchestra.console:Power off for smithi155 completed 2020-03-11T22:21:09.148 INFO:teuthology.orchestra.console:Power on smithi155 2020-03-11T22:21:09.149 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2020-03-11T22:21:09.170 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:13.198 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi155.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2020-03-11T22:21:13.407 INFO:teuthology.orchestra.console:Power on for smithi155 completed 2020-03-11T22:21:13.508 INFO:teuthology.provision.fog.smithi155:Waiting for deploy to finish 2020-03-11T22:24:14.907 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:23.959 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:33.016 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:42.072 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:24:51.130 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:00.190 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:09.274 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:18.334 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:27.418 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:25:36.478 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:00.922 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:09.979 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:10.318 INFO:teuthology.orchestra.run.smithi155:> hostname 2020-03-11T22:26:10.666 INFO:teuthology.orchestra.run.smithi155.stdout:smithi136 2020-03-11T22:26:10.666 INFO:teuthology.orchestra.run.smithi155:> grep smithi136 /etc/hosts 2020-03-11T22:26:10.681 INFO:teuthology.orchestra.run.smithi155.stdout:172.21.15.136 smithi136.front.sepia.ceph.com smithi136 2020-03-11T22:26:10.681 INFO:teuthology.orchestra.run.smithi155:> sudo hostname smithi155 2020-03-11T22:26:10.799 INFO:teuthology.orchestra.run.smithi155:> sudo sed -i -e 's/smithi136/smithi155/g' /etc/hosts 2020-03-11T22:26:10.826 INFO:teuthology.orchestra.run.smithi155:> sudo sed -i -e 's/172.21.15.136/172.21.15.155/g' /etc/hosts 2020-03-11T22:26:10.893 INFO:teuthology.orchestra.run.smithi155:> sudo sed -i -e 's/smithi136/smithi155/g' /etc/hostname 2020-03-11T22:26:10.960 INFO:teuthology.orchestra.run.smithi155:> sudo hostname smithi155 2020-03-11T22:26:11.026 INFO:teuthology.provision.fog.smithi155:Deploy complete! 2020-03-11T22:26:11.028 DEBUG:teuthology.parallel:result is None 2020-03-11T22:26:11.137 INFO:teuthology.lock.ops:Checking smithi155.front.sepia.ceph.com 2020-03-11T22:26:11.138 INFO:teuthology.lock.ops:New key found. Updating... 2020-03-11T22:26:11.181 INFO:teuthology.lock.ops:Updating [smithi155.front.sepia.ceph.com]: set os type and version on server 2020-03-11T22:26:11.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:11.182 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:11.249 INFO:teuthology.orchestra.run.smithi155:> true 2020-03-11T22:26:11.419 INFO:teuthology.orchestra.run.smithi155:> uname -m 2020-03-11T22:26:11.474 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2020-03-11T22:26:11.474 INFO:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2020-03-11T22:26:11.527 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Linux" 2020-03-11T22:26:11.528 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="8 (Core)" 2020-03-11T22:26:11.528 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2020-03-11T22:26:11.528 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:26:11.528 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="8" 2020-03-11T22:26:11.528 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:26:11.528 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:26:11.528 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2020-03-11T22:26:11.529 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:26:11.529 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:26:11.529 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:26:11.529 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:26:11.529 INFO:teuthology.orchestra.run.smithi155.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:26:11.529 INFO:teuthology.orchestra.run.smithi155.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:26:11.529 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:26:11.529 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:26:11.530 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:26:11.530 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2020-03-11T22:26:11.553 INFO:teuthology.task.internal.lock_machines:1 smithi machines locked this try, 1/1 locked so far 2020-03-11T22:26:11.572 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi155.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDUU0/5oOPYsDL37Zip/TtabXRM1sX9H6yT3c4/bPnOQlkUS3g8MXxowNi6xqxKGW4ghAflkRyUkHMyr2KldW66OKZtCY1dsajJjslpNl7XDQZt6jeGGG8x59UzkGX6XuaAeJpGA4JIzNDbokjJXIEvUYYlFkfP6sbntokvXT/ZgmujFm4MQ+iFg351Vnll7TLXvaTyeXq8peEqqYJ/+FOBblK8dzav1yIRL1nks/XRXw5489jiGYWAAvmO1/F6sQDpkJTio0b/Pk9lBpSyYSc8qozfwHMaJFSYRwg8zZJxLctEL3rhC8AZVeWUtsFNB44mQ2nUehBu27bBQiry1cWRlteCjB+CRQHTIaNAFf3N0QK/zjcgTeiLTz6VrBXJ/QTdaq1ryNw0MvN4vxwvTgSY5RiJDBXHDmrG5w4piLkf+ra+Cvuky/DffF2yU/4dA7ezrnNS7RaULMBEBIsknTHZls3h4aKlkq7yaI7jQ2rpCHSDKja7GFJQ/PTpPL4x1v0= 2020-03-11T22:26:11.572 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2020-03-11T22:26:11.617 INFO:teuthology.run_tasks:Running task internal.save_config... 2020-03-11T22:26:11.621 INFO:teuthology.task.internal:Saving configuration 2020-03-11T22:26:11.631 INFO:teuthology.run_tasks:Running task internal.check_lock... 2020-03-11T22:26:11.640 INFO:teuthology.task.internal.check_lock:Checking locks... 2020-03-11T22:26:11.657 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2020-03-11 22:20:55.936297', u'locked_by': u'scheduled_dzafman@bionic-laptop', u'up': True, u'mac_address': None, u'name': u'smithi155.front.sepia.ceph.com', u'os_version': u'8', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBpzYurrKaZAdqhUWH417FNf9tB6yiOT/H37Z38bplwKLdDMBJgFoopdfHLHwMtKGFMclQ0kkw3QWCiePdQerV8=', u'description': u'/home/teuthworker/archive/dzafman-2020-03-11_15:13:33-rados-wip-zafman-octo-diag-distro-basic-smithi/4848251'} 2020-03-11T22:26:11.657 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2020-03-11T22:26:11.666 INFO:teuthology.task.internal:roles: ubuntu@smithi155.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2020-03-11T22:26:11.666 INFO:teuthology.run_tasks:Running task console_log... 2020-03-11T22:26:11.716 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2020-03-11T22:26:11.717 INFO:teuthology.run_tasks:Running task internal.connect... 2020-03-11T22:26:11.726 INFO:teuthology.task.internal:Opening connections... 2020-03-11T22:26:11.726 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:26:11.727 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:26:11.800 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2020-03-11T22:26:11.808 INFO:teuthology.orchestra.run.smithi155:> uname -m 2020-03-11T22:26:11.992 INFO:teuthology.orchestra.run.smithi155.stdout:x86_64 2020-03-11T22:26:11.992 INFO:teuthology.orchestra.run.smithi155:> cat /etc/os-release 2020-03-11T22:26:12.046 INFO:teuthology.orchestra.run.smithi155.stdout:NAME="CentOS Linux" 2020-03-11T22:26:12.047 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION="8 (Core)" 2020-03-11T22:26:12.047 INFO:teuthology.orchestra.run.smithi155.stdout:ID="centos" 2020-03-11T22:26:12.047 INFO:teuthology.orchestra.run.smithi155.stdout:ID_LIKE="rhel fedora" 2020-03-11T22:26:12.047 INFO:teuthology.orchestra.run.smithi155.stdout:VERSION_ID="8" 2020-03-11T22:26:12.047 INFO:teuthology.orchestra.run.smithi155.stdout:PLATFORM_ID="platform:el8" 2020-03-11T22:26:12.047 INFO:teuthology.orchestra.run.smithi155.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2020-03-11T22:26:12.047 INFO:teuthology.orchestra.run.smithi155.stdout:ANSI_COLOR="0;31" 2020-03-11T22:26:12.047 INFO:teuthology.orchestra.run.smithi155.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2020-03-11T22:26:12.048 INFO:teuthology.orchestra.run.smithi155.stdout:HOME_URL="https://www.centos.org/" 2020-03-11T22:26:12.048 INFO:teuthology.orchestra.run.smithi155.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2020-03-11T22:26:12.048 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:26:12.048 INFO:teuthology.orchestra.run.smithi155.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2020-03-11T22:26:12.048 INFO:teuthology.orchestra.run.smithi155.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2020-03-11T22:26:12.048 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2020-03-11T22:26:12.048 INFO:teuthology.orchestra.run.smithi155.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2020-03-11T22:26:12.049 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:26:12.049 INFO:teuthology.lock.ops:Updating smithi155.front.sepia.ceph.com on lock server 2020-03-11T22:26:12.070 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2020-03-11T22:26:12.097 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2020-03-11T22:26:12.106 INFO:teuthology.task.internal:Checking for old test directory... 2020-03-11T22:26:12.106 INFO:teuthology.orchestra.run.smithi155:> test '!' -e /home/ubuntu/cephtest 2020-03-11T22:26:12.121 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2020-03-11T22:26:12.130 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2020-03-11T22:26:12.131 INFO:teuthology.orchestra.run.smithi155:> test -z $(ls -A /var/lib/ceph) 2020-03-11T22:26:12.177 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2020-03-11T22:26:12.207 INFO:teuthology.run_tasks:Running task kernel... 2020-03-11T22:26:12.229 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}} 2020-03-11T22:26:12.229 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}}, timeout 300 2020-03-11T22:26:12.229 INFO:teuthology.orchestra.run.smithi155:> uname -r 2020-03-11T22:26:12.244 INFO:teuthology.orchestra.run.smithi155.stdout:4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:26:12.245 INFO:teuthology.task.kernel:Running kernel on smithi155: 4.18.0-147.5.1.el8_1.x86_64 2020-03-11T22:26:12.245 INFO:teuthology.orchestra.run.smithi155:> sudo yum install -y kernel 2020-03-11T22:26:15.102 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1 day, 7:00:22 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:26:15.696 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-4.18.0-147.el8.x86_64 is already installed. 2020-03-11T22:26:15.697 INFO:teuthology.orchestra.run.smithi155.stdout:Package kernel-4.18.0-147.5.1.el8_1.x86_64 is already installed. 2020-03-11T22:26:15.879 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:26:15.880 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2020-03-11T22:26:15.880 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:26:16.059 INFO:teuthology.orchestra.run.smithi155:> echo no | sudo yum reinstall kernel || true 2020-03-11T22:26:16.484 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1 day, 7:00:23 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:26:16.885 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:26:16.886 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:26:16.886 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2020-03-11T22:26:16.886 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:26:16.886 INFO:teuthology.orchestra.run.smithi155.stdout:Reinstalling: 2020-03-11T22:26:16.886 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:26:16.886 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:26:16.887 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:26:16.887 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:26:16.887 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:26:16.887 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:26:16.903 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.1 M 2020-03-11T22:26:16.903 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 0 2020-03-11T22:26:16.904 INFO:teuthology.orchestra.run.smithi155.stderr:Operation aborted. 2020-03-11T22:26:17.064 INFO:teuthology.orchestra.run.smithi155.stdout:Is this ok [y/N]: 2020-03-11T22:26:17.064 INFO:teuthology.orchestra.run.smithi155:> sudo yum reinstall -y kernel || true 2020-03-11T22:26:17.498 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 1 day, 7:00:24 ago on Tue 10 Mar 2020 03:25:53 PM UTC. 2020-03-11T22:26:18.222 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:26:18.223 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:26:18.223 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2020-03-11T22:26:18.223 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:26:18.223 INFO:teuthology.orchestra.run.smithi155.stdout:Reinstalling: 2020-03-11T22:26:18.223 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 4.18.0-147.5.1.el8_1 CentOS-Base 1.5 M 2020-03-11T22:26:18.223 INFO:teuthology.orchestra.run.smithi155.stdout: kernel x86_64 4.18.0-147.el8 CentOS-Base 1.5 M 2020-03-11T22:26:18.223 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:26:18.224 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:26:18.224 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:26:18.224 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:26:18.224 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 3.1 M 2020-03-11T22:26:18.225 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 0 2020-03-11T22:26:18.225 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:26:18.329 INFO:teuthology.orchestra.run.smithi155.stdout:(1/2): kernel-4.18.0-147.5.1.el8_1.x86_64.rpm 25 MB/s | 1.5 MB 00:00 2020-03-11T22:26:18.344 INFO:teuthology.orchestra.run.smithi155.stdout:(2/2): kernel-4.18.0-147.el8.x86_64.rpm 20 MB/s | 1.5 MB 00:00 2020-03-11T22:26:18.346 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:26:18.346 INFO:teuthology.orchestra.run.smithi155.stdout:Total 27 MB/s | 3.1 MB 00:00 2020-03-11T22:26:18.376 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:26:18.625 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:26:18.625 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:26:18.756 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:26:18.756 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:26:19.323 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:26:19.634 INFO:teuthology.orchestra.run.smithi155.stdout: Reinstalling : kernel-4.18.0-147.el8.x86_64 1/4 2020-03-11T22:26:19.875 INFO:teuthology.orchestra.run.smithi155.stdout: Reinstalling : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:26:19.875 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:26:20.084 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:26:20.085 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:26:20.391 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: kernel-4.18.0-147.5.1.el8_1.x86_64 4/4 2020-03-11T22:26:20.391 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 1/4 2020-03-11T22:26:20.391 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-4.18.0-147.5.1.el8_1.x86_64 2/4 2020-03-11T22:26:20.391 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 3/4 2020-03-11T22:26:20.769 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : kernel-4.18.0-147.el8.x86_64 4/4 2020-03-11T22:26:20.769 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:26:20.769 INFO:teuthology.orchestra.run.smithi155.stdout:Reinstalled: 2020-03-11T22:26:20.769 INFO:teuthology.orchestra.run.smithi155.stdout: kernel-4.18.0-147.5.1.el8_1.x86_64 kernel-4.18.0-147.el8.x86_64 2020-03-11T22:26:20.770 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:26:20.770 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:26:21.643 INFO:teuthology.orchestra.run.smithi155:> rpm -q kernel --last | head -n 1 2020-03-11T22:26:21.700 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:26:19 PM UTC 2020-03-11T22:26:21.702 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:26:21.702 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-147.5.1.el8_1.x86_64 Expected: 4.18.0-147.el8.x86_64 2020-03-11T22:26:21.702 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2020-03-11T22:26:21.702 DEBUG:teuthology.task.kernel:src is distro, skipping download 2020-03-11T22:26:21.702 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2020-03-11T22:26:21.702 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi155.front.sepia.ceph.com, path=None, version=distro) 2020-03-11T22:26:21.703 INFO:teuthology.orchestra.run.smithi155:> rpm -q kernel --last | head -n 1 2020-03-11T22:26:21.745 INFO:teuthology.orchestra.run.smithi155.stdout:kernel-4.18.0-147.el8.x86_64 Wed 11 Mar 2020 10:26:19 PM UTC 2020-03-11T22:26:21.746 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-147.el8.x86_64 2020-03-11T22:26:21.747 INFO:teuthology.orchestra.run.smithi155:> sudo rpm -qi grub2-tools 2020-03-11T22:26:21.819 INFO:teuthology.orchestra.run.smithi155.stdout:Name : grub2-tools 2020-03-11T22:26:21.820 INFO:teuthology.orchestra.run.smithi155.stdout:Epoch : 1 2020-03-11T22:26:21.820 INFO:teuthology.orchestra.run.smithi155.stdout:Version : 2.02 2020-03-11T22:26:21.820 INFO:teuthology.orchestra.run.smithi155.stdout:Release : 78.el8 2020-03-11T22:26:21.820 INFO:teuthology.orchestra.run.smithi155.stdout:Architecture: x86_64 2020-03-11T22:26:21.820 INFO:teuthology.orchestra.run.smithi155.stdout:Install Date: Tue 10 Mar 2020 03:09:09 PM UTC 2020-03-11T22:26:21.820 INFO:teuthology.orchestra.run.smithi155.stdout:Group : System Environment/Base 2020-03-11T22:26:21.820 INFO:teuthology.orchestra.run.smithi155.stdout:Size : 9192691 2020-03-11T22:26:21.821 INFO:teuthology.orchestra.run.smithi155.stdout:License : GPLv3+ 2020-03-11T22:26:21.821 INFO:teuthology.orchestra.run.smithi155.stdout:Signature : RSA/SHA256, Wed 04 Dec 2019 11:50:58 PM UTC, Key ID 05b555b38483c65d 2020-03-11T22:26:21.821 INFO:teuthology.orchestra.run.smithi155.stdout:Source RPM : grub2-2.02-78.el8.src.rpm 2020-03-11T22:26:21.821 INFO:teuthology.orchestra.run.smithi155.stdout:Build Date : Mon 25 Nov 2019 04:08:30 PM UTC 2020-03-11T22:26:21.821 INFO:teuthology.orchestra.run.smithi155.stdout:Build Host : kbuilder.bsys.centos.org 2020-03-11T22:26:21.821 INFO:teuthology.orchestra.run.smithi155.stdout:Relocations : (not relocatable) 2020-03-11T22:26:21.821 INFO:teuthology.orchestra.run.smithi155.stdout:Packager : CentOS BuildSystem 2020-03-11T22:26:21.821 INFO:teuthology.orchestra.run.smithi155.stdout:Vendor : CentOS 2020-03-11T22:26:21.822 INFO:teuthology.orchestra.run.smithi155.stdout:URL : http://www.gnu.org/software/grub/ 2020-03-11T22:26:21.822 INFO:teuthology.orchestra.run.smithi155.stdout:Summary : Support tools for GRUB. 2020-03-11T22:26:21.822 INFO:teuthology.orchestra.run.smithi155.stdout:Description : 2020-03-11T22:26:21.822 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:26:21.822 INFO:teuthology.orchestra.run.smithi155.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2020-03-11T22:26:21.822 INFO:teuthology.orchestra.run.smithi155.stdout:customizable bootloader with modular architecture. It supports a rich 2020-03-11T22:26:21.822 INFO:teuthology.orchestra.run.smithi155.stdout:variety of kernel formats, file systems, computer architectures and 2020-03-11T22:26:21.822 INFO:teuthology.orchestra.run.smithi155.stdout:hardware devices. 2020-03-11T22:26:21.823 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:26:21.823 INFO:teuthology.orchestra.run.smithi155.stdout:This subpackage provides tools for support of all platforms. 2020-03-11T22:26:21.824 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2020-03-11T22:26:21.824 INFO:teuthology.task.kernel:Updating grub on smithi155 to boot 4.18.0-147.el8.x86_64 2020-03-11T22:26:21.824 INFO:teuthology.orchestra.run.smithi155:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2020-03-11T22:26:22.595 INFO:teuthology.orchestra.run.smithi155.stderr:Generating grub configuration file ... 2020-03-11T22:26:24.847 INFO:teuthology.orchestra.run.smithi155.stderr:done 2020-03-11T22:26:24.849 INFO:teuthology.orchestra.run.smithi155:> mktemp 2020-03-11T22:26:24.862 INFO:teuthology.orchestra.run.smithi155.stdout:/tmp/tmp.iVxJCqYAE9 2020-03-11T22:26:24.862 INFO:teuthology.orchestra.run.smithi155:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iVxJCqYAE9 2020-03-11T22:26:24.951 INFO:teuthology.orchestra.run.smithi155:> sudo chmod 0666 /tmp/tmp.iVxJCqYAE9 2020-03-11T22:26:25.060 DEBUG:teuthology.orchestra.remote:smithi155:/tmp/tmp.iVxJCqYAE9 is 4KB 2020-03-11T22:26:25.108 INFO:teuthology.orchestra.run.smithi155:> rm -fr /tmp/tmp.iVxJCqYAE9 2020-03-11T22:26:25.122 INFO:teuthology.orchestra.run.smithi155:> sudo /bin/ls /boot/loader/entries || true 2020-03-11T22:26:25.196 INFO:teuthology.orchestra.run.smithi155.stdout:291b34331a614822a44a2b4be2021771-0-rescue.conf 2020-03-11T22:26:25.196 INFO:teuthology.orchestra.run.smithi155.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.5.1.el8_1.x86_64.conf 2020-03-11T22:26:25.196 INFO:teuthology.orchestra.run.smithi155.stdout:291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64.conf 2020-03-11T22:26:25.197 INFO:teuthology.orchestra.run.smithi155:> sudo grub2-set-default 291b34331a614822a44a2b4be2021771-4.18.0-147.el8.x86_64 2020-03-11T22:26:25.349 INFO:teuthology.orchestra.run.smithi155:> sudo shutdown -r now 2020-03-11T22:26:55.370 INFO:teuthology.misc:Re-opening connections... 2020-03-11T22:26:55.370 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:26:55.371 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:26:55.371 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:13.766 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:13.766 DEBUG:teuthology.misc:waited 18.3956639767 2020-03-11T22:27:14.767 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:14.767 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:14.767 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:17.834 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:17.834 DEBUG:teuthology.misc:waited 22.4634590149 2020-03-11T22:27:18.834 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:18.834 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:18.835 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:20.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:20.902 DEBUG:teuthology.misc:waited 25.5315160751 2020-03-11T22:27:21.903 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:21.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:21.903 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:23.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:23.974 DEBUG:teuthology.misc:waited 28.6035070419 2020-03-11T22:27:24.975 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:24.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:24.975 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:27.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:27.046 DEBUG:teuthology.misc:waited 31.675508976 2020-03-11T22:27:28.047 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:28.047 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:28.047 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:30.118 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:30.118 DEBUG:teuthology.misc:waited 34.7474739552 2020-03-11T22:27:31.118 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:31.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:31.119 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:33.190 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:33.190 DEBUG:teuthology.misc:waited 37.8195559978 2020-03-11T22:27:34.190 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:34.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:34.191 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:36.262 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:36.262 DEBUG:teuthology.misc:waited 40.8914940357 2020-03-11T22:27:37.263 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:37.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:37.263 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:39.334 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:39.334 DEBUG:teuthology.misc:waited 43.9635438919 2020-03-11T22:27:40.334 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:40.335 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:40.335 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:42.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:42.406 DEBUG:teuthology.misc:waited 47.0355269909 2020-03-11T22:27:43.407 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:43.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:43.407 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:45.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:45.478 DEBUG:teuthology.misc:waited 50.107585907 2020-03-11T22:27:46.479 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:46.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:46.480 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:48.550 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:48.550 DEBUG:teuthology.misc:waited 53.1795299053 2020-03-11T22:27:49.551 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:49.551 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:49.552 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:27:51.626 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.155 2020-03-11T22:27:51.626 DEBUG:teuthology.misc:waited 56.2556478977 2020-03-11T22:27:52.627 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:27:52.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:27:52.627 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:28:52.629 DEBUG:teuthology.orchestra.remote:timed out 2020-03-11T22:28:52.629 DEBUG:teuthology.misc:waited 117.258976936 2020-03-11T22:28:53.631 INFO:teuthology.misc:trying to connect to ubuntu@smithi155.front.sepia.ceph.com 2020-03-11T22:28:53.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:28:53.631 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:28:53.976 INFO:teuthology.orchestra.run.smithi155:> true 2020-03-11T22:28:54.294 DEBUG:teuthology.misc:waited 118.92377305 2020-03-11T22:28:55.295 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2020-03-11T22:28:55.295 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-147.el8.x86_64"... 2020-03-11T22:28:55.296 INFO:teuthology.orchestra.run.smithi155:> uname -r 2020-03-11T22:28:55.312 INFO:teuthology.orchestra.run.smithi155.stdout:4.18.0-147.el8.x86_64 2020-03-11T22:28:55.312 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-147.el8.x86_64 vs 4.18.0-147.el8.x86_64 2020-03-11T22:28:55.312 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2020-03-11T22:28:56.313 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2020-03-11T22:28:56.313 INFO:teuthology.orchestra.run.smithi155:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2020-03-11T22:28:56.402 INFO:teuthology.orchestra.run.smithi155.stdout:ttyS1 2020-03-11T22:28:56.414 INFO:teuthology.run_tasks:Running task internal.base... 2020-03-11T22:28:56.424 INFO:teuthology.task.internal:Creating test directory... 2020-03-11T22:28:56.424 INFO:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2020-03-11T22:28:56.440 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2020-03-11T22:28:56.449 INFO:teuthology.run_tasks:Running task internal.archive... 2020-03-11T22:28:56.463 INFO:teuthology.task.internal:Creating archive directory... 2020-03-11T22:28:56.463 INFO:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2020-03-11T22:28:56.515 INFO:teuthology.run_tasks:Running task internal.coredump... 2020-03-11T22:28:56.524 INFO:teuthology.task.internal:Enabling coredump saving... 2020-03-11T22:28:56.525 INFO:teuthology.orchestra.run.smithi155:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2020-03-11T22:28:56.585 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:28:56.599 INFO:teuthology.orchestra.run.smithi155.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:28:56.600 INFO:teuthology.run_tasks:Running task internal.sudo... 2020-03-11T22:28:56.610 INFO:teuthology.task.internal:Configuring sudo... 2020-03-11T22:28:56.610 INFO:teuthology.orchestra.run.smithi155:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2020-03-11T22:28:56.670 INFO:teuthology.run_tasks:Running task internal.syslog... 2020-03-11T22:28:56.681 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2020-03-11T22:28:56.682 INFO:teuthology.orchestra.run.smithi155:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2020-03-11T22:28:56.724 INFO:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:28:56.798 INFO:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2020-03-11T22:28:56.855 INFO:teuthology.orchestra.run.smithi155:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:28:56.927 INFO:teuthology.orchestra.run.smithi155:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2020-03-11T22:28:56.978 INFO:teuthology.orchestra.run.smithi155:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2020-03-11T22:28:57.063 INFO:teuthology.orchestra.run.smithi155:> sudo service rsyslog restart 2020-03-11T22:28:57.173 INFO:teuthology.orchestra.run.smithi155.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2020-03-11T22:28:57.486 INFO:teuthology.run_tasks:Running task internal.timer... 2020-03-11T22:28:57.497 INFO:teuthology.task.internal:Starting timer... 2020-03-11T22:28:57.497 INFO:teuthology.run_tasks:Running task pcp... 2020-03-11T22:28:57.571 INFO:teuthology.run_tasks:Running task selinux... 2020-03-11T22:28:57.585 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2020-03-11T22:28:57.586 INFO:teuthology.orchestra.run.smithi155:> sudo service auditd rotate 2020-03-11T22:28:57.639 INFO:teuthology.orchestra.run.smithi155.stdout:Rotating logs: [ OK ] 2020-03-11T22:28:57.641 DEBUG:teuthology.task.selinux:Getting current SELinux state 2020-03-11T22:28:57.641 INFO:teuthology.orchestra.run.smithi155:> /usr/sbin/getenforce 2020-03-11T22:28:57.658 INFO:teuthology.orchestra.run.smithi155.stdout:Permissive 2020-03-11T22:28:57.659 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi155.front.sepia.ceph.com': 'permissive'} 2020-03-11T22:28:57.659 INFO:teuthology.orchestra.run.smithi155:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|scontext=system_u:system_r:logrotate_t:s0\)' 2020-03-11T22:28:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:28:57.727 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2020-03-11T22:28:57.727 INFO:teuthology.orchestra.run.smithi155:> sudo /usr/sbin/setenforce permissive 2020-03-11T22:28:57.812 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2020-03-11T22:28:57.827 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2020-03-11T22:28:57.849 INFO:teuthology.repo_utils:Fetching master from origin 2020-03-11T22:28:58.095 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2020-03-11T22:28:58.223 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2020-03-11T22:28:58.228 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi155.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2020-03-11T22:28:59.090 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2020-03-11T22:29:01.154 INFO:teuthology.task.ansible.out: [WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2020-03-11T22:29:04.555 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2020-03-11T22:29:04.596 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:04 +0000 (0:00:00.164) 0:00:00.164 ******* 2020-03-11T22:29:07.354 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi155.front.sepia.ceph.com] 2020-03-11T22:29:07.437 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:07 +0000 (0:00:02.840) 0:00:03.005 ******* 2020-03-11T22:29:07.762 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2020-03-11T22:29:07.763 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:29:07.804 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:07 +0000 (0:00:00.366) 0:00:03.372 ******* 2020-03-11T22:29:08.369 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2020-03-11T22:29:08.370 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2020-03-11T22:29:08.434 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:08 +0000 (0:00:00.629) 0:00:04.002 ******* 2020-03-11T22:29:08.460 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2020-03-11T22:29:08.462 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:08.500 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:08 +0000 (0:00:00.065) 0:00:04.068 ******* 2020-03-11T22:29:08.830 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2020-03-11T22:29:08.831 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:29:08.872 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:08 +0000 (0:00:00.371) 0:00:04.439 ******* 2020-03-11T22:29:09.891 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2020-03-11T22:29:09.893 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1583965748.89-252635841449432/source", "state": "file", "uid": 0} 2020-03-11T22:29:09.928 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:09 +0000 (0:00:01.056) 0:00:05.496 ******* 2020-03-11T22:29:10.404 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2020-03-11T22:29:10.405 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2020-03-11T22:29:10.407 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2020-03-11T22:29:10.512 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2020-03-11T22:29:10.521 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2020-03-11T22:29:10.728 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:10 +0000 (0:00:00.800) 0:00:06.296 ******* 2020-03-11T22:29:11.392 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi155.front.sepia.ceph.com] 2020-03-11T22:29:11.479 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2020-03-11T22:29:11.480 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:11 +0000 (0:00:00.750) 0:00:07.047 ******* 2020-03-11T22:29:12.261 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2020-03-11T22:29:12.263 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.428781", "end": "2020-03-11 22:29:12.235615", "rc": 0, "start": "2020-03-11 22:29:11.806834", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:29:12.298 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:12 +0000 (0:00:00.819) 0:00:07.866 ******* 2020-03-11T22:29:12.509 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2020-03-11T22:29:12.511 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010748", "end": "2020-03-11 22:29:12.488342", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:29:12.477594", "stderr": "", "stderr_lines": [], "stdout": "d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6", "stdout_lines": ["d6cc316c8d11cadac2a03d4f84b215a35cf6c4b6"]} 2020-03-11T22:29:12.552 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:12 +0000 (0:00:00.253) 0:00:08.120 ******* 2020-03-11T22:29:12.589 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2020-03-11T22:29:12.590 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} 2020-03-11T22:29:12.630 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:12 +0000 (0:00:00.077) 0:00:08.197 ******* 2020-03-11T22:29:12.656 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2020-03-11T22:29:12.657 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:12.695 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:12 +0000 (0:00:00.065) 0:00:08.263 ******* 2020-03-11T22:29:12.721 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2020-03-11T22:29:12.722 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:12.759 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:12 +0000 (0:00:00.063) 0:00:08.326 ******* 2020-03-11T22:29:12.785 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2020-03-11T22:29:12.786 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:12.822 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:12 +0000 (0:00:00.063) 0:00:08.390 ******* 2020-03-11T22:29:12.984 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2020-03-11T22:29:12.985 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.048 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:13 +0000 (0:00:00.225) 0:00:08.615 ******* 2020-03-11T22:29:13.074 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2020-03-11T22:29:13.075 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.114 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:13 +0000 (0:00:00.066) 0:00:08.682 ******* 2020-03-11T22:29:13.160 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2020-03-11T22:29:13.161 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.172 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.173 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.194 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.196 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.197 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.208 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.210 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.221 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.233 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.233 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.244 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.245 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.256 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.267 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.268 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.279 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.291 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.291 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.303 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.303 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.315 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.326 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.327 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.338 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.339 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.349 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.350 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.361 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.372 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.373 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.375 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.385 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.385 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.396 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.407 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.418 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.429 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.429 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.441 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.441 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.452 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.453 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.463 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.474 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.475 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.486 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.487 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.497 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.508 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.509 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.520 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.530 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.541 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.542 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.553 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.553 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.565 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.577 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.578 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.589 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.590 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.601 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.601 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.612 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.623 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.625 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.635 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.635 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.646 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.647 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.658 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.659 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.670 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.670 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.682 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.683 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.694 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.694 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.706 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.707 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.718 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.729 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.731 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.732 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.743 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.744 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.755 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon BgvhtHMrCcelT98ekPDMoA 6fa3cf08186b93b966b68841f3507a0809d348c5097b39b6140832cac2afd511"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.755 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.767 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.768 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.778 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.779 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.790 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.790 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.801 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.813 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.814 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.815 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.826 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.826 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.837 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.848 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.850 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.860 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f', u'name': u'toabctl'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.861 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39', u'name': u'mgfritch'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.872 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d', u'name': u'xiubli'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "xiubli@fedora1 vosB15bA3uALkBrmjcKFUg 4ef5cb132c30563fd5fb9e4e9352786ad0d2258e41ed63f91c1f65f69b54d50d"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.873 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f', u'name': u'jafaj'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.884 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef', u'name': u'jschmid'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.886 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518', u'name': u'kachau'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad GsZIGCdLojpvOZDxwcWSlg 0638220db65953e0cd1f134c499e4a713b32a882c8a92458afcb495dbd087518"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.895 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034', u'name': u'khiremat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@hrk-thinkpad 6DWhbAZZDUo789PultFaJw f586e852c2dab5f56e3e12237dadd8f76f61d9136479329698e6f7dcd91f9034"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.897 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:13.908 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'ovpn': u'kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23', u'name': u'kapandya'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad elavnUlTLKzuE7/eJ2nxMw 4ac31f4c197b2b6325a404950175faeedc105ea45092672e924b6edafe90af23"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:13.909 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:13.962 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:13 +0000 (0:00:00.848) 0:00:09.530 ******* 2020-03-11T22:29:13.990 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2020-03-11T22:29:14.039 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:00.076) 0:00:09.607 ******* 2020-03-11T22:29:14.065 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.067 INFO:teuthology.task.ansible.out:TASK [users : Merge managed_users and managed_admin users] ********************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.106 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:00.067) 0:00:09.674 ******* 2020-03-11T22:29:14.108 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.164 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.166 INFO:teuthology.task.ansible.out:TASK [users : Clone the keys repo] ********************************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.203 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:00.097) 0:00:09.771 ******* 2020-03-11T22:29:14.205 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.230 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.231 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.270 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:00.067) 0:00:09.838 ******* 2020-03-11T22:29:14.272 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.297 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.298 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys for each user with literal keys] ********** 2020-03-11T22:29:14.299 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.338 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:00.067) 0:00:09.906 ******* 2020-03-11T22:29:14.340 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.366 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.367 INFO:teuthology.task.ansible.out:TASK [users : Filter the revoked_users list] *********************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.404 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:00.065) 0:00:09.971 ******* 2020-03-11T22:29:14.404 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.441 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.443 INFO:teuthology.task.ansible.out:TASK [users : Remove revoked users] ******************************************** skipping: [smithi155.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.444 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.455 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.457 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.458 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.469 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.471 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.481 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.483 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.495 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.496 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.498 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.508 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.509 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.520 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.521 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.531 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.533 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.534 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.545 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.546 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.548 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.559 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.560 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.571 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.571 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.573 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.583 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.585 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.586 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.597 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.598 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:14.600 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.601 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.612 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.614 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.625 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.626 INFO:teuthology.task.ansible.out: skipping: [smithi155.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.628 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.638 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.640 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.650 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.661 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.672 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.672 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:14.717 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:14 +0000 (0:00:00.312) 0:00:10.284 ******* 2020-03-11T22:29:15.539 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2020-03-11T22:29:15.539 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "9c97a4e7e2fa12022ca70c847609415d5b1cb1c9", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "state": "file", "uid": 0} 2020-03-11T22:29:15.578 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:15 +0000 (0:00:00.861) 0:00:11.146 ******* 2020-03-11T22:29:15.614 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2020-03-11T22:29:15.616 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => { "msg": "Host smithi155.front.sepia.ceph.com is running CentOS 8.1 (Core)" } 2020-03-11T22:29:15.655 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:15 +0000 (0:00:00.077) 0:00:11.223 ******* 2020-03-11T22:29:15.702 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2020-03-11T22:29:15.704 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2020-03-11T22:29:15.740 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:15 +0000 (0:00:00.084) 0:00:11.308 ******* 2020-03-11T22:29:15.768 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2020-03-11T22:29:15.769 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:15.806 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:15 +0000 (0:00:00.065) 0:00:11.373 ******* 2020-03-11T22:29:16.078 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2020-03-11T22:29:16.080 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.073504", "end": "2020-03-11 22:29:16.054393", "rc": 0, "start": "2020-03-11 22:29:15.980889", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2020-03-11T22:29:16.127 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.321) 0:00:11.695 ******* 2020-03-11T22:29:16.154 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2020-03-11T22:29:16.155 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:16.199 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.071) 0:00:11.766 ******* 2020-03-11T22:29:16.225 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2020-03-11T22:29:16.227 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:16.263 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.064) 0:00:11.831 ******* 2020-03-11T22:29:16.289 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2020-03-11T22:29:16.291 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:16.331 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.068) 0:00:11.899 ******* 2020-03-11T22:29:16.357 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2020-03-11T22:29:16.358 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:16.410 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.078) 0:00:11.978 ******* 2020-03-11T22:29:16.436 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2020-03-11T22:29:16.437 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:16.475 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.065) 0:00:12.043 ******* 2020-03-11T22:29:16.502 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2020-03-11T22:29:16.505 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:16.552 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.077) 0:00:12.120 ******* 2020-03-11T22:29:16.591 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2020-03-11T22:29:16.592 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:29:16.631 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.078) 0:00:12.199 ******* 2020-03-11T22:29:16.658 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2020-03-11T22:29:16.659 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:16.697 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.065) 0:00:12.265 ******* 2020-03-11T22:29:16.733 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2020-03-11T22:29:16.734 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:16.778 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.081) 0:00:12.346 ******* 2020-03-11T22:29:16.815 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2020-03-11T22:29:16.816 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:29:16.858 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.079) 0:00:12.425 ******* 2020-03-11T22:29:16.884 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2020-03-11T22:29:16.885 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:16.931 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:16 +0000 (0:00:00.073) 0:00:12.499 ******* 2020-03-11T22:29:16.973 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2020-03-11T22:29:16.974 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:29:17.026 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.094) 0:00:12.594 ******* 2020-03-11T22:29:17.052 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2020-03-11T22:29:17.054 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.096 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.069) 0:00:12.663 ******* 2020-03-11T22:29:17.123 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2020-03-11T22:29:17.125 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.161 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.065) 0:00:12.729 ******* 2020-03-11T22:29:17.187 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2020-03-11T22:29:17.189 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.231 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.069) 0:00:12.799 ******* 2020-03-11T22:29:17.257 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2020-03-11T22:29:17.258 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.299 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.068) 0:00:12.867 ******* 2020-03-11T22:29:17.326 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2020-03-11T22:29:17.326 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:29:17.367 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.067) 0:00:12.935 ******* 2020-03-11T22:29:17.394 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:29:17.395 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.434 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.067) 0:00:13.002 ******* 2020-03-11T22:29:17.471 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2020-03-11T22:29:17.472 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.513 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.078) 0:00:13.081 ******* 2020-03-11T22:29:17.539 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2020-03-11T22:29:17.539 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.579 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.066) 0:00:13.147 ******* 2020-03-11T22:29:17.616 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2020-03-11T22:29:17.616 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.660 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.080) 0:00:13.228 ******* 2020-03-11T22:29:17.697 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.738 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.077) 0:00:13.306 ******* 2020-03-11T22:29:17.765 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:17.805 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.066) 0:00:13.372 ******* 2020-03-11T22:29:17.832 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** skipping: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2020-03-11T22:29:17.870 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.065) 0:00:13.437 ******* 2020-03-11T22:29:17.897 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2020-03-11T22:29:17.937 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:17 +0000 (0:00:00.067) 0:00:13.505 ******* 2020-03-11T22:29:17.974 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2020-03-11T22:29:17.974 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:18.013 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.076) 0:00:13.581 ******* 2020-03-11T22:29:18.258 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:29:18.298 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:18 +0000 (0:00:00.284) 0:00:13.866 ******* 2020-03-11T22:29:18.813 INFO:teuthology.task.ansible.out: 2020-03-11T22:29:18.815 INFO:teuthology.task.ansible.out:TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** ok: [smithi155.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2da7f101e29cfe8f3b630dbd6da82b4f9a749fee", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/8/Everything/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 304, "state": "file", "uid": 0} 2020-03-11T22:29:19.280 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "ddfaa38799b2dd4c23933a375aa4c3ff61bf7625", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/8/Everything/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 338, "state": "file", "uid": 0} 2020-03-11T22:29:19.326 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:19 +0000 (0:00:01.027) 0:00:14.894 ******* 2020-03-11T22:29:19.876 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2020-03-11T22:29:19.877 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2020-03-11T22:29:20.347 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2020-03-11T22:29:20.390 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:01.063) 0:00:15.958 ******* 2020-03-11T22:29:20.417 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2020-03-11T22:29:20.419 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.459 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.069) 0:00:16.027 ******* 2020-03-11T22:29:20.485 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:29:20.487 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.537 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.077) 0:00:16.105 ******* 2020-03-11T22:29:20.562 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2020-03-11T22:29:20.564 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.600 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.063) 0:00:16.168 ******* 2020-03-11T22:29:20.626 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2020-03-11T22:29:20.627 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.663 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.062) 0:00:16.231 ******* 2020-03-11T22:29:20.689 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2020-03-11T22:29:20.690 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.725 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.061) 0:00:16.293 ******* 2020-03-11T22:29:20.751 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2020-03-11T22:29:20.752 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.787 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.061) 0:00:16.355 ******* 2020-03-11T22:29:20.823 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2020-03-11T22:29:20.824 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.860 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.072) 0:00:16.427 ******* 2020-03-11T22:29:20.887 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2020-03-11T22:29:20.888 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.922 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.062) 0:00:16.490 ******* 2020-03-11T22:29:20.947 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2020-03-11T22:29:20.949 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:20.985 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:20 +0000 (0:00:00.063) 0:00:16.553 ******* 2020-03-11T22:29:21.528 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2020-03-11T22:29:21.529 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.345833", "end": "2020-03-11 22:29:21.500334", "rc": 0, "start": "2020-03-11 22:29:21.154501", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:21.568 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:21 +0000 (0:00:00.583) 0:00:17.136 ******* 2020-03-11T22:29:24.952 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2020-03-11T22:29:24.953 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: krb5-workstation"]} 2020-03-11T22:29:25.000 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:24 +0000 (0:00:03.428) 0:00:20.564 ******* 2020-03-11T22:29:25.025 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.061 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.064) 0:00:20.629 ******* 2020-03-11T22:29:25.087 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2020-03-11T22:29:25.088 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.128 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.067) 0:00:20.696 ******* 2020-03-11T22:29:25.154 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2020-03-11T22:29:25.156 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.191 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.063) 0:00:20.759 ******* 2020-03-11T22:29:25.228 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2020-03-11T22:29:25.229 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:25.270 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.078) 0:00:20.838 ******* 2020-03-11T22:29:25.767 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2020-03-11T22:29:25.768 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2020-03-11T22:29:25.819 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:25 +0000 (0:00:00.548) 0:00:21.387 ******* 2020-03-11T22:29:26.742 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2020-03-11T22:29:26.743 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: smartmontools"]} 2020-03-11T22:29:26.787 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:26 +0000 (0:00:00.968) 0:00:22.355 ******* 2020-03-11T22:29:27.332 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2020-03-11T22:29:27.332 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2020-03-11T22:29:27.849 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2020-03-11T22:29:28.307 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2020-03-11T22:29:28.349 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:28 +0000 (0:00:01.561) 0:00:23.917 ******* 2020-03-11T22:29:28.555 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2020-03-11T22:29:28.556 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:29:28.606 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:28 +0000 (0:00:00.257) 0:00:24.174 ******* 2020-03-11T22:29:29.102 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2020-03-11T22:29:29.103 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2020-03-11T22:29:29.560 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2020-03-11T22:29:30.017 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2020-03-11T22:29:30.073 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:30 +0000 (0:00:01.466) 0:00:25.641 ******* 2020-03-11T22:29:30.099 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2020-03-11T22:29:30.101 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:30.156 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:30 +0000 (0:00:00.082) 0:00:25.724 ******* 2020-03-11T22:29:30.194 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2020-03-11T22:29:30.196 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2020-03-11T22:29:30.241 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:30 +0000 (0:00:00.084) 0:00:25.809 ******* 2020-03-11T22:29:30.442 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2020-03-11T22:29:30.443 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2020-03-11T22:29:30.479 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:30 +0000 (0:00:00.237) 0:00:26.047 ******* 2020-03-11T22:29:30.536 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2020-03-11T22:29:30.537 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:30.548 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:30.559 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2020-03-11T22:29:30.599 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:30 +0000 (0:00:00.119) 0:00:26.167 ******* 2020-03-11T22:29:31.516 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2020-03-11T22:29:31.517 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2020-03-11T22:29:31.553 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:31 +0000 (0:00:00.954) 0:00:27.121 ******* 2020-03-11T22:29:31.579 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2020-03-11T22:29:31.580 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:31.616 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:31 +0000 (0:00:00.062) 0:00:27.183 ******* 2020-03-11T22:29:32.094 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2020-03-11T22:29:32.095 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2020-03-11T22:29:32.132 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.515) 0:00:27.699 ******* 2020-03-11T22:29:32.622 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2020-03-11T22:29:32.623 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2020-03-11T22:29:32.661 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.528) 0:00:28.228 ******* 2020-03-11T22:29:32.687 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2020-03-11T22:29:32.689 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:32.726 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.065) 0:00:28.294 ******* 2020-03-11T22:29:32.952 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2020-03-11T22:29:32.953 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:29:32.989 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:32 +0000 (0:00:00.263) 0:00:28.557 ******* 2020-03-11T22:29:33.242 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2020-03-11T22:29:33.244 INFO:teuthology.task.ansible.out:fatal: [smithi155.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2020-03-11T22:29:33.285 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:33 +0000 (0:00:00.296) 0:00:28.853 ******* 2020-03-11T22:29:33.313 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2020-03-11T22:29:33.315 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:29:33.354 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:33 +0000 (0:00:00.068) 0:00:28.922 ******* 2020-03-11T22:29:33.870 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2020-03-11T22:29:33.911 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:33 +0000 (0:00:00.556) 0:00:29.479 ******* 2020-03-11T22:29:34.766 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2020-03-11T22:29:34.768 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:28:21 UTC", "ActiveEnterTimestampMonotonic": "12251839", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount system.slice var-run.mount basic.target tmp.mount local-fs.target systemd-tmpfiles-setup.service nss-lookup.target network.target time-sync.target systemd-journald.socket sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:28:21 UTC", "AssertTimestampMonotonic": "12250945", "Before": "plymouth-quit.service xdm.service shutdown.target multi-user.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:28:21 UTC", "ConditionTimestampMonotonic": "12250944", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "D 2020-03-11T22:29:34.769 INFO:teuthology.task.ansible.out:efaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2397", "ExecMainStartTimestamp": "Wed 2020-03-11 22:28:21 UTC", "ExecMainStartTimestampMonotonic": "12251791", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[Wed 2020-03-11 22:28:21 UTC] ; stop_time=[n/a] ; pid=2397 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "986", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:28:21 UTC", "InactiveExitTimestampMonotonic": "12251839", "InvocationID": "ff08ad4062a9412eb1aa2b5d826e302f", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "Limi 2020-03-11T22:29:34.770 INFO:teuthology.task.ansible.out:tDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127437", "LimitNPROCSoft": "127437", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127437", "LimitSIGPENDINGSoft": "127437", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2397", "MemoryAccounting": "yes", "MemoryCurrent": "1982464", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "n 2020-03-11T22:29:34.772 INFO:teuthology.task.ansible.out:rpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:28:21 UTC", "StateChangeTimestampMonotonic": "12251839", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "990", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:28:21 UTC", "WatchdogTimestampMonotonic": "12251838", "WatchdogUSec": "0"}} 2020-03-11T22:29:34.814 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:34 +0000 (0:00:00.902) 0:00:30.381 ******* 2020-03-11T22:29:35.028 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2020-03-11T22:29:35.029 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001692", "end": "2020-03-11 22:29:35.002886", "rc": 0, "start": "2020-03-11 22:29:35.001194", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2020-03-11T22:29:35.066 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:35 +0000 (0:00:00.252) 0:00:30.634 ******* 2020-03-11T22:29:35.974 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2020-03-11T22:29:35.976 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils"]} 2020-03-11T22:29:36.015 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.949) 0:00:31.583 ******* 2020-03-11T22:29:36.931 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2020-03-11T22:29:36.933 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: selinux-policy-targeted"]} 2020-03-11T22:29:36.990 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:36 +0000 (0:00:00.974) 0:00:32.558 ******* 2020-03-11T22:29:38.580 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2020-03-11T22:29:38.582 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2020-03-11T22:29:38.632 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:38 +0000 (0:00:01.642) 0:00:34.200 ******* 2020-03-11T22:29:55.065 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2020-03-11T22:29:55.066 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.231559", "end": "2020-03-11 22:29:55.039100", "failed_when_result": false, "rc": 0, "start": "2020-03-11 22:29:38.807541", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2020-03-11T22:29:55.102 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:55 +0000 (0:00:16.469) 0:00:50.670 ******* 2020-03-11T22:29:55.604 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2020-03-11T22:29:55.606 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2020-03-11T22:29:55.644 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:55 +0000 (0:00:00.541) 0:00:51.211 ******* 2020-03-11T22:29:55.856 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2020-03-11T22:29:55.857 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.008919", "end": "2020-03-11 22:29:55.827420", "rc": 0, "start": "2020-03-11 22:29:55.818501", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:55.894 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:55 +0000 (0:00:00.250) 0:00:51.462 ******* 2020-03-11T22:29:56.094 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2020-03-11T22:29:56.096 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.002126", "end": "2020-03-11 22:29:56.068193", "rc": 0, "start": "2020-03-11 22:29:56.066067", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:29:56.132 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:29:56 +0000 (0:00:00.238) 0:00:51.700 ******* 2020-03-11T22:30:12.434 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2020-03-11T22:30:12.436 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.068541", "end": "2020-03-11 22:30:12.403474", "rc": 0, "start": "2020-03-11 22:29:56.334933", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:30:12.471 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:12 +0000 (0:00:16.339) 0:01:08.039 ******* 2020-03-11T22:30:12.683 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2020-03-11T22:30:12.684 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2020-03-11T22:30:12.722 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:12 +0000 (0:00:00.250) 0:01:08.290 ******* 2020-03-11T22:30:12.758 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2020-03-11T22:30:12.759 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2020-03-11T22:30:12.796 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:12 +0000 (0:00:00.074) 0:01:08.364 ******* 2020-03-11T22:30:12.842 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2020-03-11T22:30:12.844 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:30:12.881 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:12 +0000 (0:00:00.084) 0:01:08.449 ******* 2020-03-11T22:30:12.918 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2020-03-11T22:30:12.919 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "copr_repos": ["ktdreyer/ceph-el8"], "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["@core", "@base", "dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2020-03-11T22:30:12.957 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:12 +0000 (0:00:00.075) 0:01:08.525 ******* 2020-03-11T22:30:12.993 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2020-03-11T22:30:12.995 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2020-03-11T22:30:13.032 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:13 +0000 (0:00:00.075) 0:01:08.600 ******* 2020-03-11T22:30:13.226 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2020-03-11T22:30:13.228 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2020-03-11T22:30:13.264 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:13 +0000 (0:00:00.231) 0:01:08.831 ******* 2020-03-11T22:30:13.457 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2020-03-11T22:30:13.458 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "name": "ubuntu", "state": "present", "system": false} 2020-03-11T22:30:13.495 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:13 +0000 (0:00:00.231) 0:01:09.062 ******* 2020-03-11T22:30:13.520 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2020-03-11T22:30:13.522 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:13.557 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:13 +0000 (0:00:00.061) 0:01:09.124 ******* 2020-03-11T22:30:13.582 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2020-03-11T22:30:13.583 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:13.622 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:13 +0000 (0:00:00.065) 0:01:09.190 ******* 2020-03-11T22:30:13.838 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2020-03-11T22:30:13.840 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2020-03-11T22:30:13.876 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:13 +0000 (0:00:00.253) 0:01:09.444 ******* 2020-03-11T22:30:14.535 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2020-03-11T22:30:14.536 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2020-03-11T22:30:14.573 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:14 +0000 (0:00:00.697) 0:01:10.141 ******* 2020-03-11T22:30:14.609 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2020-03-11T22:30:14.611 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi155"}, "changed": false} 2020-03-11T22:30:14.645 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:14 +0000 (0:00:00.071) 0:01:10.213 ******* 2020-03-11T22:30:15.668 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2020-03-11T22:30:15.669 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi155.front.sepia.ceph.com", "ansible_hostname": "smithi155", "ansible_nodename": "smithi155"}, "changed": true, "name": "smithi155"} 2020-03-11T22:30:15.706 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:15 +0000 (0:00:01.061) 0:01:11.274 ******* 2020-03-11T22:30:16.178 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2020-03-11T22:30:16.178 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2020-03-11T22:30:16.220 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:16 +0000 (0:00:00.513) 0:01:11.788 ******* 2020-03-11T22:30:16.709 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2020-03-11T22:30:16.710 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2020-03-11T22:30:16.748 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:16 +0000 (0:00:00.527) 0:01:12.316 ******* 2020-03-11T22:30:17.434 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2020-03-11T22:30:17.435 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2020-03-11T22:30:17.474 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:17 +0000 (0:00:00.726) 0:01:13.042 ******* 2020-03-11T22:30:17.964 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2020-03-11T22:30:17.966 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2020-03-11T22:30:18.011 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:18 +0000 (0:00:00.536) 0:01:13.579 ******* 2020-03-11T22:30:18.213 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:30:18.215 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:30:18.249 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:18 +0000 (0:00:00.238) 0:01:13.817 ******* 2020-03-11T22:30:18.451 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:30:18.452 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2020-03-11T22:30:18.488 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:18 +0000 (0:00:00.238) 0:01:14.056 ******* 2020-03-11T22:30:18.714 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:30:18.715 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1211, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2020-03-11T22:30:18.753 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:18 +0000 (0:00:00.264) 0:01:14.321 ******* 2020-03-11T22:30:19.234 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:30:19.235 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2020-03-11T22:30:19.272 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:19 +0000 (0:00:00.519) 0:01:14.840 ******* 2020-03-11T22:30:19.749 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:30:19.749 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2020-03-11T22:30:19.788 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:19 +0000 (0:00:00.516) 0:01:15.356 ******* 2020-03-11T22:30:21.631 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:30:21.633 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.634758", "end": "2020-03-11 22:30:21.598973", "rc": 0, "start": "2020-03-11 22:30:19.964215", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:30:21.669 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:21 +0000 (0:00:01.881) 0:01:17.237 ******* 2020-03-11T22:30:22.186 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2020-03-11T22:30:22.186 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "53b24162baf9c2fcf22da2b5c2c258a91997ffad", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 425, "state": "file", "uid": 0} 2020-03-11T22:30:22.651 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3e4763db3d6c8e2d723452e76e0b9ed9030ed28d", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 408, "state": "file", "uid": 0} 2020-03-11T22:30:23.108 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "6d4b75787c58f146a27a5ee2b688dd525283abf1", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 410, "state": "file", "uid": 0} 2020-03-11T22:30:23.148 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:23 +0000 (0:00:01.478) 0:01:18.716 ******* 2020-03-11T22:30:23.651 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* ok: [smithi155.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2020-03-11T22:30:23.693 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:23 +0000 (0:00:00.545) 0:01:19.261 ******* 2020-03-11T22:30:24.192 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2020-03-11T22:30:24.192 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', u'name': u'CentOS-$releasever - Base'}, 'key': u'CentOS-Base'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2020-03-11T22:30:24.640 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', u'name': u'CentOS-$releasever - AppStream'}, 'key': u'CentOS-AppStream'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2020-03-11T22:30:24.642 INFO:teuthology.task.ansible.out: 2020-03-11T22:30:25.097 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 1, u'gpgkey': u'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial', u'enabled': 1, u'mirrorlist': u'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', u'name': u'CentOS-$releasever - Extras'}, 'key': u'CentOS-Extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2020-03-11T22:30:25.148 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:25 +0000 (0:00:01.454) 0:01:20.716 ******* 2020-03-11T22:30:25.677 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2020-03-11T22:30:25.678 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item=ktdreyer/ceph-el8) => {"ansible_loop_var": "item", "changed": true, "cmd": ["dnf", "-y", "copr", "enable", "ktdreyer/ceph-el8"], "delta": "0:00:00.321494", "end": "2020-03-11 22:30:25.653108", "item": "ktdreyer/ceph-el8", "rc": 0, "start": "2020-03-11 22:30:25.331614", "stderr": "", "stderr_lines": [], "stdout": "Repository successfully enabled.", "stdout_lines": ["Repository successfully enabled."], "warnings": ["Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message."]} 2020-03-11T22:30:25.679 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the dnf module rather than running 'dnf'. If you need to use command because dnf is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. 2020-03-11T22:30:25.732 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:25 +0000 (0:00:00.583) 0:01:21.300 ******* 2020-03-11T22:30:25.758 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2020-03-11T22:30:25.759 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:25.810 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:25 +0000 (0:00:00.065) 0:01:21.366 ******* 2020-03-11T22:30:25.834 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2020-03-11T22:30:25.836 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:25.847 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:25.884 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:25 +0000 (0:00:00.085) 0:01:21.451 ******* 2020-03-11T22:30:25.921 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:25.967 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:25 +0000 (0:00:00.083) 0:01:21.535 ******* 2020-03-11T22:30:26.004 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:30:26.006 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2020-03-11T22:30:26.040 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:26 +0000 (0:00:00.072) 0:01:21.607 ******* 2020-03-11T22:30:26.076 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2020-03-11T22:30:26.077 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:26.112 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:26 +0000 (0:00:00.072) 0:01:21.680 ******* 2020-03-11T22:30:26.148 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2020-03-11T22:30:26.150 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:26.185 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:26 +0000 (0:00:00.073) 0:01:21.753 ******* 2020-03-11T22:30:26.395 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2020-03-11T22:30:26.396 INFO:teuthology.task.ansible.out:fatal: [smithi155.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006699", "end": "2020-03-11 22:30:26.375145", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:30:26.368446", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2020-03-11T22:30:26.433 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:26 +0000 (0:00:00.248) 0:01:22.001 ******* 2020-03-11T22:30:26.469 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2020-03-11T22:30:26.471 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:26.507 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:26 +0000 (0:00:00.073) 0:01:22.075 ******* 2020-03-11T22:30:29.113 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:30:29.114 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64", "Removed: librbd1-1:12.2.7-9.el8.x86_64"]} 2020-03-11T22:30:29.153 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:29 +0000 (0:00:02.645) 0:01:24.721 ******* 2020-03-11T22:30:32.152 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:30:32.154 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64", "Removed: hdparm-9.54-2.el8.x86_64", "Removed: tuned-2.12.0-3.el8_1.1.noarch"]} 2020-03-11T22:30:32.189 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:32 +0000 (0:00:03.036) 0:01:27.757 ******* 2020-03-11T22:30:37.550 INFO:teuthology.task.ansible.out: TASK [testnode : Base group workaround for BZ1782899] ************************** 2020-03-11T22:30:37.551 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": "dnf -y groupinstall base --nobest && dnf -y group upgrade base", "delta": "0:00:05.104066", "end": "2020-03-11 22:30:37.529430", "rc": 0, "start": "2020-03-11 22:30:32.425364", "stderr": "No match for group package \"insights-client\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"centos-release-eula\"\nNo match for group package \"insights-client\"", "stderr_lines": ["No match for group package \"insights-client\"", "No match for group package \"centos-release-eula\"", "No match for group package \"centos-release-eula\"", "No match for group package \"insights-client\""], "stdout": "CentOS-8 - AppStream 142 kB/s | 4.3 kB 00:00 \nCentOS-8 - Base 131 kB/s | 3.8 kB 00:00 \nCentOS-8 - Extras 53 kB/s | 1.5 kB 00:00 \nCopr repo for ceph-el8 owned by ktdreyer 69 kB/s | 3.3 kB 00:00 \nExtra Packages for Enterprise Linux 165 kB/s | 5.3 kB 00:00 \nlab-extras 1.2 MB/s | 2.9 kB 00:00 \nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n================================================================================\nInstalling Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!\nLast metadata expiration check: 0:00:02 ago on Wed 11 Mar 2020 10:30:33 PM UTC.\nDependencies resolved.\n================================================================================\n Package Architecture Version Repository Size\n============================================================= 2020-03-11T22:30:37.553 INFO:teuthology.task.ansible.out:===================\nUpgrading Groups:\n Base \n\nTransaction Summary\n================================================================================\n\nComplete!", "stdout_lines": ["CentOS-8 - AppStream 142 kB/s | 4.3 kB 00:00 ", "CentOS-8 - Base 131 kB/s | 3.8 kB 00:00 ", "CentOS-8 - Extras 53 kB/s | 1.5 kB 00:00 ", "Copr repo for ceph-el8 owned by ktdreyer 69 kB/s | 3.3 kB 00:00 ", "Extra Packages for Enterprise Linux 165 kB/s | 5.3 kB 00:00 ", "lab-extras 1.2 MB/s | 2.9 kB 00:00 ", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Installing Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!", "Last metadata expiration check: 0:00:02 ago on Wed 11 Mar 2020 10:30:33 PM UTC.", "Dependencies resolved.", "================================================================================", " Package Architecture Version Repository Size", "================================================================================", "Upgrading Groups:", " Base ", "", "Transaction Summary", "================================================================================", "", "Complete!"]} 2020-03-11T22:30:37.590 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:37 +0000 (0:00:05.400) 0:01:33.158 ******* 2020-03-11T22:30:45.450 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:30:45.451 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Group core installed.", "Group base installed.", "Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: tuned-2.12.0-3.el8_1.1.noarch", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+266+ba744077.2.x86_64", "Installed: boost-program-options-1.66.0-6.el8.x86_64", "Installed: boost-random-1.66.0-6.el8.x86_64", "Installed: hdparm-9.54-2.el8.x86_64"]} 2020-03-11T22:30:45.487 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:45 +0000 (0:00:07.897) 0:01:41.055 ******* 2020-03-11T22:30:45.514 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2020-03-11T22:30:45.515 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:45.551 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:45 +0000 (0:00:00.063) 0:01:41.119 ******* 2020-03-11T22:30:46.489 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:30:46.490 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2020-03-11T22:30:46.525 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:46 +0000 (0:00:00.973) 0:01:42.093 ******* 2020-03-11T22:30:46.561 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:30:46.563 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:46.600 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:46 +0000 (0:00:00.074) 0:01:42.167 ******* 2020-03-11T22:30:46.926 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2020-03-11T22:30:46.928 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.socket basic.target sysinit.target dbus.service system.slice polkit.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "iptables.service ebtables.service shutdown.target ipset.service ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTim 2020-03-11T22:30:46.929 INFO:teuthology.task.ansible.out:estampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "16777216", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127437", "LimitNPROCSoft": "127437", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127437", 2020-03-11T22:30:46.931 INFO:teuthology.task.ansible.out:"LimitSIGPENDINGSoft": "127437", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "dbus.socket sysinit.target system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3 2020-03-11T22:30:46.932 INFO:teuthology.task.ansible.out:", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2020-03-11T22:30:46.967 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:46 +0000 (0:00:00.367) 0:01:42.535 ******* 2020-03-11T22:30:47.263 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2020-03-11T22:30:47.264 INFO:teuthology.task.ansible.out:fatal: [smithi155.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2020-03-11T22:30:47.300 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:47 +0000 (0:00:00.333) 0:01:42.868 ******* 2020-03-11T22:30:48.052 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2020-03-11T22:30:48.054 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2020-03-11T22:30:48.092 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.791) 0:01:43.659 ******* 2020-03-11T22:30:48.117 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2020-03-11T22:30:48.119 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:48.155 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.063) 0:01:43.723 ******* 2020-03-11T22:30:48.181 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2020-03-11T22:30:48.219 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.063) 0:01:43.787 ******* 2020-03-11T22:30:48.245 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2020-03-11T22:30:48.247 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:48.284 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.064) 0:01:43.852 ******* 2020-03-11T22:30:48.310 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2020-03-11T22:30:48.312 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:48.350 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.065) 0:01:43.918 ******* 2020-03-11T22:30:48.388 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2020-03-11T22:30:48.389 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:48.428 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.077) 0:01:43.996 ******* 2020-03-11T22:30:48.464 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2020-03-11T22:30:48.467 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:48.501 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.073) 0:01:44.069 ******* 2020-03-11T22:30:48.535 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2020-03-11T22:30:48.573 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.072) 0:01:44.141 ******* 2020-03-11T22:30:48.600 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2020-03-11T22:30:48.601 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:48.659 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.085) 0:01:44.227 ******* 2020-03-11T22:30:48.695 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2020-03-11T22:30:48.697 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:48.735 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.075) 0:01:44.303 ******* 2020-03-11T22:30:48.761 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2020-03-11T22:30:48.762 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:48.799 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.064) 0:01:44.367 ******* 2020-03-11T22:30:48.827 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2020-03-11T22:30:48.828 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:48.870 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.071) 0:01:44.438 ******* 2020-03-11T22:30:48.899 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:48.939 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:48 +0000 (0:00:00.067) 0:01:44.506 ******* 2020-03-11T22:30:48.966 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:30:48.967 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.005 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.066) 0:01:44.573 ******* 2020-03-11T22:30:49.031 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2020-03-11T22:30:49.032 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.069 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.063) 0:01:44.637 ******* 2020-03-11T22:30:49.095 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.132 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.063) 0:01:44.700 ******* 2020-03-11T22:30:49.158 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2020-03-11T22:30:49.159 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.199 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.063) 0:01:44.764 ******* 2020-03-11T22:30:49.222 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2020-03-11T22:30:49.224 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.262 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.065) 0:01:44.830 ******* 2020-03-11T22:30:49.288 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2020-03-11T22:30:49.290 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.327 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.065) 0:01:44.895 ******* 2020-03-11T22:30:49.353 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2020-03-11T22:30:49.354 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.391 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.063) 0:01:44.959 ******* 2020-03-11T22:30:49.417 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2020-03-11T22:30:49.419 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.455 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.063) 0:01:45.023 ******* 2020-03-11T22:30:49.491 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2020-03-11T22:30:49.493 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.534 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.078) 0:01:45.101 ******* 2020-03-11T22:30:49.559 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2020-03-11T22:30:49.561 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.597 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.063) 0:01:45.165 ******* 2020-03-11T22:30:49.623 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2020-03-11T22:30:49.625 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.667 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.069) 0:01:45.235 ******* 2020-03-11T22:30:49.703 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2020-03-11T22:30:49.705 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.751 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.084) 0:01:45.319 ******* 2020-03-11T22:30:49.779 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2020-03-11T22:30:49.781 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.817 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.065) 0:01:45.385 ******* 2020-03-11T22:30:49.843 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2020-03-11T22:30:49.844 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.882 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.064) 0:01:45.450 ******* 2020-03-11T22:30:49.908 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2020-03-11T22:30:49.910 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:49.947 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:49 +0000 (0:00:00.065) 0:01:45.515 ******* 2020-03-11T22:30:49.973 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2020-03-11T22:30:49.975 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:50.011 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:50 +0000 (0:00:00.063) 0:01:45.579 ******* 2020-03-11T22:30:50.037 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2020-03-11T22:30:50.038 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:50.075 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:50 +0000 (0:00:00.064) 0:01:45.643 ******* 2020-03-11T22:30:50.103 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2020-03-11T22:30:50.104 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:50.140 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:50 +0000 (0:00:00.064) 0:01:45.708 ******* 2020-03-11T22:30:50.166 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2020-03-11T22:30:50.168 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:50.204 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:50 +0000 (0:00:00.064) 0:01:45.772 ******* 2020-03-11T22:30:50.231 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:50.268 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:50 +0000 (0:00:00.063) 0:01:45.836 ******* 2020-03-11T22:30:50.295 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:30:50.296 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:50.332 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:50 +0000 (0:00:00.063) 0:01:45.900 ******* 2020-03-11T22:30:50.554 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:30:50.555 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2020-03-11T22:30:50.593 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:50 +0000 (0:00:00.260) 0:01:46.161 ******* 2020-03-11T22:30:50.805 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:30:50.806 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:30:50.845 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:50 +0000 (0:00:00.251) 0:01:46.413 ******* 2020-03-11T22:30:51.065 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:30:51.066 INFO:teuthology.task.ansible.out: [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2020-03-11T22:30:51.103 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:51 +0000 (0:00:00.257) 0:01:46.670 ******* 2020-03-11T22:30:51.129 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:30:51.131 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:51.172 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:51 +0000 (0:00:00.069) 0:01:46.740 ******* 2020-03-11T22:30:51.203 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:30:51.204 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:51.246 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:51 +0000 (0:00:00.073) 0:01:46.814 ******* 2020-03-11T22:30:51.272 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2020-03-11T22:30:51.273 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:51.312 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:51 +0000 (0:00:00.065) 0:01:46.880 ******* 2020-03-11T22:30:51.338 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:30:51.339 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:51.385 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:51 +0000 (0:00:00.072) 0:01:46.953 ******* 2020-03-11T22:30:51.421 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:30:51.422 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:51.461 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:51 +0000 (0:00:00.076) 0:01:47.029 ******* 2020-03-11T22:30:51.488 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:30:51.489 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:51.534 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:51 +0000 (0:00:00.072) 0:01:47.102 ******* 2020-03-11T22:30:51.561 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2020-03-11T22:30:51.562 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:51.602 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:51 +0000 (0:00:00.068) 0:01:47.170 ******* 2020-03-11T22:30:51.628 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2020-03-11T22:30:51.630 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:51.670 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:51 +0000 (0:00:00.067) 0:01:47.238 ******* 2020-03-11T22:30:51.697 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2020-03-11T22:30:51.698 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:51.745 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:51 +0000 (0:00:00.074) 0:01:47.313 ******* 2020-03-11T22:30:51.771 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2020-03-11T22:30:51.772 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:51.813 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:51 +0000 (0:00:00.067) 0:01:47.381 ******* 2020-03-11T22:30:51.964 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2020-03-11T22:30:51.965 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:52.098 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.284) 0:01:47.665 ******* 2020-03-11T22:30:52.123 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2020-03-11T22:30:52.125 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:52.160 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.062) 0:01:47.728 ******* 2020-03-11T22:30:52.197 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2020-03-11T22:30:52.198 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:52.235 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.074) 0:01:47.803 ******* 2020-03-11T22:30:52.261 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2020-03-11T22:30:52.262 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:52.298 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.062) 0:01:47.866 ******* 2020-03-11T22:30:52.324 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2020-03-11T22:30:52.325 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:52.369 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.071) 0:01:47.937 ******* 2020-03-11T22:30:52.395 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2020-03-11T22:30:52.397 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:52.438 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.069) 0:01:48.006 ******* 2020-03-11T22:30:52.478 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:52.513 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.074) 0:01:48.081 ******* 2020-03-11T22:30:52.539 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2020-03-11T22:30:52.540 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:52.576 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.062) 0:01:48.144 ******* 2020-03-11T22:30:52.612 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2020-03-11T22:30:52.613 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:52.651 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.074) 0:01:48.219 ******* 2020-03-11T22:30:52.677 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2020-03-11T22:30:52.678 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:52.716 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.065) 0:01:48.284 ******* 2020-03-11T22:30:52.753 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:52.794 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.077) 0:01:48.361 ******* 2020-03-11T22:30:52.820 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2020-03-11T22:30:52.821 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:52.857 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.063) 0:01:48.425 ******* 2020-03-11T22:30:52.894 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:30:52.896 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2020-03-11T22:30:52.943 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:52 +0000 (0:00:00.086) 0:01:48.511 ******* 2020-03-11T22:30:52.992 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2020-03-11T22:30:52.993 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:53.005 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2020-03-11T22:30:53.016 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:53.027 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:53.039 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:53.040 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:53.051 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2020-03-11T22:30:53.114 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:53 +0000 (0:00:00.170) 0:01:48.682 ******* 2020-03-11T22:30:53.296 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2020-03-11T22:30:53.298 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.005005", "end": "2020-03-11 22:30:53.275161", "rc": 0, "start": "2020-03-11 22:30:53.270156", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2020-03-11T22:30:53.341 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:53 +0000 (0:00:00.226) 0:01:48.909 ******* 2020-03-11T22:30:53.368 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2020-03-11T22:30:53.370 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:53.420 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:53 +0000 (0:00:00.079) 0:01:48.988 ******* 2020-03-11T22:30:53.457 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2020-03-11T22:30:53.459 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:53.499 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:53 +0000 (0:00:00.078) 0:01:49.067 ******* 2020-03-11T22:30:53.526 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2020-03-11T22:30:53.527 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:53.568 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:53 +0000 (0:00:00.068) 0:01:49.136 ******* 2020-03-11T22:30:53.595 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2020-03-11T22:30:53.596 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:53.641 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:53 +0000 (0:00:00.072) 0:01:49.208 ******* 2020-03-11T22:30:54.582 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2020-03-11T22:30:54.583 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper"]} 2020-03-11T22:30:54.643 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:54 +0000 (0:00:00.995) 0:01:50.204 ******* 2020-03-11T22:30:54.674 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:30:54.675 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item={u'block_used': 12964179, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130929, u'size_available': 930328285184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989861, u'device': u'/dev/sda1', u'inode_used': 65115, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130929, "block_size": 4096, "block_total": 240095108, "block_used": 12964179, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989861, "inode_total": 61054976, "inode_used": 65115, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328285184, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}} 2020-03-11T22:30:54.714 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:54 +0000 (0:00:00.077) 0:01:50.281 ******* 2020-03-11T22:30:54.925 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2020-03-11T22:30:54.926 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005225", "end": "2020-03-11 22:30:54.903224", "rc": 0, "start": "2020-03-11 22:30:54.897999", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:30:54.970 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:54 +0000 (0:00:00.256) 0:01:50.538 ******* 2020-03-11T22:30:55.017 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2020-03-11T22:30:55.018 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'block_used': 12964179, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130929, u'size_available': 930328285184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989861, u'device': u'/dev/sda1', u'inode_used': 65115, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130929, "block_size": 4096, "block_total": 240095108, "block_used": 12964179, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989861, "inode_total": 61054976, "inode_used": 65115, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328285184, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:55.056 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:55 +0000 (0:00:00.085) 0:01:50.623 ******* 2020-03-11T22:30:55.081 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2020-03-11T22:30:55.139 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:30:55 +0000 (0:00:00.083) 0:01:50.707 ******* 2020-03-11T22:30:55.196 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2020-03-11T22:30:55.197 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XNRW', u'wwn-0x5000c5009294102d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XNRW', u'wwn': u'0x5000c5009294102d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1', u'wwn-0x5000c5009294102d-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW", "wwn-0x5000c5009294102d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1", "wwn-0x5000c5009294102d-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5XNRW", "size": "931.51 GB", "support_discard": "0", "vend 2020-03-11T22:30:55.199 INFO:teuthology.task.ansible.out:or": "ATA", "virtual": 1, "wwn": "0x5000c5009294102d"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:30:56.746 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-TH815l-JzqV-okVp-nJ7z-abqz-cWds-KmTHn7', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN', u'nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.357009", "end": "2020-03-11 22:30:56.704397", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-TH815l-JzqV-okVp-nJ7z-abqz-cWds-KmTHn7", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN", "nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:30:55.347388", "stderr": "", "stderr_lines": [], "stdout": "Creating new GP 2020-03-11T22:30:56.747 INFO:teuthology.task.ansible.out:T entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:30:58.002 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnyFMJOYodhJ7Wr1PiPbEPx822ah7nL3OS'], u'uuids': [u'accc4776-0b04-45e4-ae29-363f74d73212']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.064720", "end": "2020-03-11 22:30:57.972124", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnyFMJOYodhJ7Wr1PiPbEPx822ah7nL3OS"], "labels": [], "masters": [], "uuids": ["accc4776-0b04-45e4-ae29-363f74d73212"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:30:56.907404", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:30:59.248 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnnshFIwYQTHpmcxOXseaYexbprPyaVjte'], u'uuids': [u'9e89cc12-3d40-4113-be4f-26c626743d70']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.052841", "end": "2020-03-11 22:30:59.220590", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnnshFIwYQTHpmcxOXseaYexbprPyaVjte"], "labels": [], "masters": [], "uuids": ["9e89cc12-3d40-4113-be4f-26c626743d70"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:30:58.167749", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:31:00.527 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOn1jFIueuS64ZHXgPOLVcVm6asfW3n3l3F'], u'uuids': [u'0b241bf9-adad-4dbe-9f18-3aad2e717ebb']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.054421", "end": "2020-03-11 22:31:00.487668", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOn1jFIueuS64ZHXgPOLVcVm6asfW3n3l3F"], "labels": [], "masters": [], "uuids": ["0b241bf9-adad-4dbe-9f18-3aad2e717ebb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:30:59.433247", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:31:01.775 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnNByLcOmXqna9H2cFsYhc8erTD5S6eb9h'], u'uuids': [u'd17baf9c-2306-46ab-8f9c-bb18fdbd853c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.050476", "end": "2020-03-11 22:31:01.745577", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnNByLcOmXqna9H2cFsYhc8erTD5S6eb9h"], "labels": [], "masters": [], "uuids": ["d17baf9c-2306-46ab-8f9c-bb18fdbd853c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:00.695101", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:31:03.054 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnLr3xP8JTjw61OesYihjGX9lyE2cs7fbe'], u'uuids': [u'56afebc0-afe4-42da-b77a-2da3920a2b47']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.076104", "end": "2020-03-11 22:31:03.021256", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnLr3xP8JTjw61OesYihjGX9lyE2cs7fbe"], "labels": [], "masters": [], "uuids": ["56afebc0-afe4-42da-b77a-2da3920a2b47"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:01.945152", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2020-03-11T22:31:03.099 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:03 +0000 (0:00:07.959) 0:01:58.666 ******* 2020-03-11T22:31:03.142 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** skipping: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XNRW', u'wwn-0x5000c5009294102d'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5XNRW', u'wwn': u'0x5000c5009294102d', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'mq-deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1', u'wwn-0x5000c5009294102d-part1'], u'uuids': [u'ebbdc914-4a8c-428b-9507-ff6e0d466a46']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW", "wwn-0x5000c5009294102d"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5XNRW-part1", "wwn-0x5000c5009294102d-part1"], "labels": [], "masters": [], "uuids": ["ebbdc914-4a8c-428b-9507-ff6e0d466a46"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsi 2020-03-11T22:31:03.143 INFO:teuthology.task.ansible.out:ze": "512", "serial": "Z1W5XNRW", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c5009294102d"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:03.375 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-TH815l-JzqV-okVp-nJ7z-abqz-cWds-KmTHn7', u'nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN', u'nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'model': u'INTEL SSDPEDMD400G4', u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [u'vg_nvme-lv_4', u'vg_nvme-lv_1', u'vg_nvme-lv_3', u'vg_nvme-lv_5', u'vg_nvme-lv_2'], u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037123", "end": "2020-03-11 22:31:03.350371", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_4", "vg_nvme-lv_1", "vg_nvme-lv_3", "vg_nvme-lv_5", "vg_nvme-lv_2"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-TH815l-JzqV-okVp-nJ7z-abqz-cWds-KmTHn7", "nvme-INTEL_SSDPEDMD400G4_CVFT623300GW400BGN", "nvme-nvme.8086-43564654363233333030475734303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:03.313248", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 11 2020-03-11T22:31:03.377 INFO:teuthology.task.ansible.out:0 MiB) copied, 0.0344966 s, 3.3 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0344966 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:03.706 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnyFMJOYodhJ7Wr1PiPbEPx822ah7nL3OS'], u'uuids': [u'accc4776-0b04-45e4-ae29-363f74d73212']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.140611", "end": "2020-03-11 22:31:03.679117", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnyFMJOYodhJ7Wr1PiPbEPx822ah7nL3OS"], "labels": [], "masters": [], "uuids": ["accc4776-0b04-45e4-ae29-363f74d73212"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:03.538506", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.137971 s, 836 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.137971 s, 836 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:04.030 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnnshFIwYQTHpmcxOXseaYexbprPyaVjte'], u'uuids': [u'9e89cc12-3d40-4113-be4f-26c626743d70']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.137297", "end": "2020-03-11 22:31:04.003825", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnnshFIwYQTHpmcxOXseaYexbprPyaVjte"], "labels": [], "masters": [], "uuids": ["9e89cc12-3d40-4113-be4f-26c626743d70"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:03.866528", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.134745 s, 856 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.134745 s, 856 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:04.361 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOn1jFIueuS64ZHXgPOLVcVm6asfW3n3l3F'], u'uuids': [u'0b241bf9-adad-4dbe-9f18-3aad2e717ebb']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.136223", "end": "2020-03-11 22:31:04.328518", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOn1jFIueuS64ZHXgPOLVcVm6asfW3n3l3F"], "labels": [], "masters": [], "uuids": ["0b241bf9-adad-4dbe-9f18-3aad2e717ebb"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:04.192295", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.133563 s, 864 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.133563 s, 864 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:04.691 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnNByLcOmXqna9H2cFsYhc8erTD5S6eb9h'], u'uuids': [u'd17baf9c-2306-46ab-8f9c-bb18fdbd853c']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.151349", "end": "2020-03-11 22:31:04.666228", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnNByLcOmXqna9H2cFsYhc8erTD5S6eb9h"], "labels": [], "masters": [], "uuids": ["d17baf9c-2306-46ab-8f9c-bb18fdbd853c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:04.514879", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14882 s, 775 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14882 s, 775 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:05.034 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnLr3xP8JTjw61OesYihjGX9lyE2cs7fbe'], u'uuids': [u'56afebc0-afe4-42da-b77a-2da3920a2b47']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'model': None, u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'holders': [], u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.147769", "end": "2020-03-11 22:31:05.005058", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-5qXQhUcyQ4GJUk53k9KQd7RaKWBO2cOnLr3xP8JTjw61OesYihjGX9lyE2cs7fbe"], "labels": [], "masters": [], "uuids": ["56afebc0-afe4-42da-b77a-2da3920a2b47"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2020-03-11 22:31:04.857289", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145157 s, 795 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145157 s, 795 MB/s"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:05.081 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:05 +0000 (0:00:01.982) 0:02:00.649 ******* 2020-03-11T22:31:05.340 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2020-03-11T22:31:05.342 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.065318", "end": "2020-03-11 22:31:05.314296", "rc": 0, "start": "2020-03-11 22:31:05.248978", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:05.380 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:05 +0000 (0:00:00.298) 0:02:00.947 ******* 2020-03-11T22:31:05.612 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2020-03-11T22:31:05.614 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.042520", "end": "2020-03-11 22:31:05.590861", "rc": 0, "start": "2020-03-11 22:31:05.548341", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:05.652 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:05 +0000 (0:00:00.271) 0:02:01.219 ******* 2020-03-11T22:31:05.678 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2020-03-11T22:31:05.719 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:05 +0000 (0:00:00.067) 0:02:01.287 ******* 2020-03-11T22:31:05.745 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2020-03-11T22:31:05.747 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:05.789 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:05 +0000 (0:00:00.070) 0:02:01.357 ******* 2020-03-11T22:31:05.815 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2020-03-11T22:31:05.817 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:05.856 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:05 +0000 (0:00:00.066) 0:02:01.424 ******* 2020-03-11T22:31:05.882 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:31:05.883 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:05.924 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:05 +0000 (0:00:00.067) 0:02:01.491 ******* 2020-03-11T22:31:05.960 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2020-03-11T22:31:05.962 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:06.017 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:06 +0000 (0:00:00.092) 0:02:01.584 ******* 2020-03-11T22:31:06.223 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2020-03-11T22:31:06.224 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2020-03-11T22:31:06.264 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:06 +0000 (0:00:00.247) 0:02:01.832 ******* 2020-03-11T22:31:06.314 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2020-03-11T22:31:06.315 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={u'block_used': 12964179, u'uuid': u'ebbdc914-4a8c-428b-9507-ff6e0d466a46', u'size_total': 983429562368, u'block_total': 240095108, u'mount': u'/', u'block_available': 227130929, u'size_available': 930328285184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60989861, u'device': u'/dev/sda1', u'inode_used': 65115, u'block_size': 4096, u'options': u'rw,seclabel,relatime'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227130929, "block_size": 4096, "block_total": 240095108, "block_used": 12964179, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60989861, "inode_total": 61054976, "inode_used": 65115, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930328285184, "size_total": 983429562368, "uuid": "ebbdc914-4a8c-428b-9507-ff6e0d466a46"}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:06.369 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:06 +0000 (0:00:00.104) 0:02:01.937 ******* 2020-03-11T22:31:06.395 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2020-03-11T22:31:06.396 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:06.437 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:06 +0000 (0:00:00.067) 0:02:02.005 ******* 2020-03-11T22:31:06.474 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:31:06.476 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:06.518 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:06 +0000 (0:00:00.080) 0:02:02.085 ******* 2020-03-11T22:31:06.543 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:31:06.545 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:06.584 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:06 +0000 (0:00:00.066) 0:02:02.152 ******* 2020-03-11T22:31:06.611 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2020-03-11T22:31:06.612 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:06.649 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:06 +0000 (0:00:00.064) 0:02:02.216 ******* 2020-03-11T22:31:06.676 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2020-03-11T22:31:06.677 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:06.714 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:06 +0000 (0:00:00.065) 0:02:02.282 ******* 2020-03-11T22:31:06.740 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2020-03-11T22:31:06.741 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:06.783 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:06 +0000 (0:00:00.069) 0:02:02.351 ******* 2020-03-11T22:31:07.390 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2020-03-11T22:31:07.392 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2020-03-11T22:31:07.430 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:07 +0000 (0:00:00.646) 0:02:02.997 ******* 2020-03-11T22:31:07.973 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2020-03-11T22:31:07.974 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:31:08.303 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:31:08.633 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:31:08.957 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:31:09.321 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2020-03-11T22:31:09.370 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:01.940) 0:02:04.938 ******* 2020-03-11T22:31:09.582 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2020-03-11T22:31:09.584 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2020-03-11T22:31:09.632 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:09 +0000 (0:00:00.261) 0:02:05.200 ******* 2020-03-11T22:31:09.692 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2020-03-11T22:31:09.693 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:09.882 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:31:10.083 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:31:10.270 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:31:10.468 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2020-03-11T22:31:10.509 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.876) 0:02:06.076 ******* 2020-03-11T22:31:10.710 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2020-03-11T22:31:10.712 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2020-03-11T22:31:10.749 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.240) 0:02:06.317 ******* 2020-03-11T22:31:10.786 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2020-03-11T22:31:10.787 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2020-03-11T22:31:10.826 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:10 +0000 (0:00:00.076) 0:02:06.394 ******* 2020-03-11T22:31:11.318 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2020-03-11T22:31:11.319 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true} 2020-03-11T22:31:11.358 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.530) 0:02:06.925 ******* 2020-03-11T22:31:11.811 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2020-03-11T22:31:11.812 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2020-03-11T22:31:11.867 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.510) 0:02:07.435 ******* 2020-03-11T22:31:11.893 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2020-03-11T22:31:11.894 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:11.942 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:11 +0000 (0:00:00.074) 0:02:07.510 ******* 2020-03-11T22:31:11.970 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2020-03-11T22:31:11.970 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.015 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.073) 0:02:07.583 ******* 2020-03-11T22:31:12.041 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2020-03-11T22:31:12.042 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.084 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.069) 0:02:07.652 ******* 2020-03-11T22:31:12.111 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2020-03-11T22:31:12.113 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.152 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.067) 0:02:07.720 ******* 2020-03-11T22:31:12.178 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2020-03-11T22:31:12.179 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.215 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.063) 0:02:07.783 ******* 2020-03-11T22:31:12.243 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2020-03-11T22:31:12.244 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.280 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.064) 0:02:07.848 ******* 2020-03-11T22:31:12.306 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2020-03-11T22:31:12.308 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.346 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.065) 0:02:07.914 ******* 2020-03-11T22:31:12.372 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2020-03-11T22:31:12.374 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.412 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.066) 0:02:07.980 ******* 2020-03-11T22:31:12.439 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2020-03-11T22:31:12.441 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.479 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.066) 0:02:08.047 ******* 2020-03-11T22:31:12.509 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2020-03-11T22:31:12.511 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.547 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.068) 0:02:08.115 ******* 2020-03-11T22:31:12.574 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2020-03-11T22:31:12.575 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:12.618 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:12 +0000 (0:00:00.070) 0:02:08.186 ******* 2020-03-11T22:31:13.111 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2020-03-11T22:31:13.112 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2020-03-11T22:31:13.152 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.533) 0:02:08.719 ******* 2020-03-11T22:31:13.457 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2020-03-11T22:31:13.459 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2020-03-11 22:28:21 UTC", "ActiveEnterTimestampMonotonic": "11793152", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target basic.target tmp.mount system.slice ntpd.service ntpdate.service -.mount systemd-journald.socket systemd-tmpfiles-setup.service sntp.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2020-03-11 22:28:20 UTC", "AssertTimestampMonotonic": "10557559", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2020-03-11 22:28:20 UTC", "ConditionTimestampMonotonic": "10557532", "ConfigurationDirectoryMode": "0755", "Conflicts": "ntpd.service shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description" 2020-03-11T22:31:13.460 INFO:teuthology.task.ansible.out:: "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2326", "ExecMainStartTimestamp": "Wed 2020-03-11 22:28:21 UTC", "ExecMainStartTimestampMonotonic": "11734245", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2020-03-11 22:28:20 UTC", "InactiveExitTimestampMonotonic": "10595473", "InvocationID": "bf3da39ac75543cc94c89ac53270ea0c", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "167772 2020-03-11T22:31:13.461 INFO:teuthology.task.ansible.out:16", "LimitMEMLOCKSoft": "16777216", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "127437", "LimitNPROCSoft": "127437", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127437", "LimitSIGPENDINGSoft": "127437", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2326", "MemoryAccounting": "yes", "MemoryCurrent": "3096576", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardE 2020-03-11T22:31:13.463 INFO:teuthology.task.ansible.out:rror": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2020-03-11 22:28:21 UTC", "StateChangeTimestampMonotonic": "11793152", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "26213", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2020-03-11 22:28:21 UTC", "WatchdogTimestampMonotonic": "11734259", "WatchdogUSec": "0"}} 2020-03-11T22:31:13.497 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.344) 0:02:09.064 ******* 2020-03-11T22:31:13.523 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2020-03-11T22:31:13.524 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:13.561 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.063) 0:02:09.128 ******* 2020-03-11T22:31:13.587 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2020-03-11T22:31:13.588 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:13.637 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.076) 0:02:09.205 ******* 2020-03-11T22:31:13.663 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2020-03-11T22:31:13.664 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:13.700 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.063) 0:02:09.268 ******* 2020-03-11T22:31:13.921 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2020-03-11T22:31:13.923 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2020-03-11T22:31:13.969 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:13 +0000 (0:00:00.269) 0:02:09.537 ******* 2020-03-11T22:31:14.466 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2020-03-11T22:31:14.467 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1211, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2020-03-11T22:31:14.509 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:14 +0000 (0:00:00.540) 0:02:10.077 ******* 2020-03-11T22:31:15.689 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2020-03-11T22:31:15.689 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2020-03-11T22:31:15.727 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:15 +0000 (0:00:01.217) 0:02:11.295 ******* 2020-03-11T22:31:15.752 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2020-03-11T22:31:15.754 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:15.789 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:15 +0000 (0:00:00.061) 0:02:11.356 ******* 2020-03-11T22:31:15.815 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2020-03-11T22:31:15.816 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2020-03-11T22:31:15.856 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:15 +0000 (0:00:00.066) 0:02:11.423 ******* 2020-03-11T22:31:16.049 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2020-03-11T22:31:16.050 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2020-03-11T22:31:16.090 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:16 +0000 (0:00:00.234) 0:02:11.658 ******* 2020-03-11T22:31:16.116 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2020-03-11T22:31:16.118 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:16.159 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:16 +0000 (0:00:00.068) 0:02:11.727 ******* 2020-03-11T22:31:16.187 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2020-03-11T22:31:16.188 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2020-03-11T22:31:16.223 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:16 +0000 (0:00:00.064) 0:02:11.791 ******* 2020-03-11T22:31:16.249 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2020-03-11T22:31:16.250 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:16.302 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:16 +0000 (0:00:00.078) 0:02:11.870 ******* 2020-03-11T22:31:16.507 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2020-03-11T22:31:16.508 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2020-03-11T22:31:16.543 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:16 +0000 (0:00:00.240) 0:02:12.111 ******* 2020-03-11T22:31:16.739 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2020-03-11T22:31:16.739 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2020-03-11T22:31:16.797 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:16 +0000 (0:00:00.254) 0:02:12.365 ******* 2020-03-11T22:31:16.825 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2020-03-11T22:31:16.827 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:16.867 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:16 +0000 (0:00:00.069) 0:02:12.434 ******* 2020-03-11T22:31:16.904 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2020-03-11T22:31:16.905 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:16.906 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:16.917 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:16.928 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:16.929 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2020-03-11T22:31:16.970 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:16 +0000 (0:00:00.102) 0:02:12.537 ******* 2020-03-11T22:31:17.005 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2020-03-11T22:31:17.007 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi155.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.018 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.018 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.030 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.068 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.098) 0:02:12.636 ******* 2020-03-11T22:31:17.096 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2020-03-11T22:31:17.098 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.146 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.077) 0:02:12.714 ******* 2020-03-11T22:31:17.182 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2020-03-11T22:31:17.184 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.230 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.077) 0:02:12.792 ******* 2020-03-11T22:31:17.250 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2020-03-11T22:31:17.251 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.292 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.068) 0:02:12.860 ******* 2020-03-11T22:31:17.319 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2020-03-11T22:31:17.320 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.363 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.070) 0:02:12.931 ******* 2020-03-11T22:31:17.391 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2020-03-11T22:31:17.392 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.431 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.068) 0:02:12.999 ******* 2020-03-11T22:31:17.457 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:31:17.458 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.497 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.065) 0:02:13.064 ******* 2020-03-11T22:31:17.525 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2020-03-11T22:31:17.526 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.564 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.067) 0:02:13.132 ******* 2020-03-11T22:31:17.600 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2020-03-11T22:31:17.602 INFO:teuthology.task.ansible.out:skipping: [smithi155.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-03-11T22:31:17.642 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:17 +0000 (0:00:00.077) 0:02:13.210 ******* 2020-03-11T22:31:17.875 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2020-03-11T22:31:17.876 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004957", "end": "2020-03-11 22:31:17.833431", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2020-03-11 22:31:17.828474", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:18.061 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003028", "end": "2020-03-11 22:31:18.025697", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2020-03-11 22:31:18.022669", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:18.268 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002890", "end": "2020-03-11 22:31:18.210666", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2020-03-11 22:31:18.207776", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:18.486 INFO:teuthology.task.ansible.out:ok: [smithi155.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.025915", "end": "2020-03-11 22:31:18.446318", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2020-03-11 22:31:18.420403", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2020-03-11T22:31:18.535 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:18 +0000 (0:00:00.893) 0:02:14.103 ******* 2020-03-11T22:31:18.572 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2020-03-11T22:31:18.611 INFO:teuthology.task.ansible.out:Wednesday 11 March 2020 22:31:18 +0000 (0:00:00.075) 0:02:14.178 ******* 2020-03-11T22:31:18.803 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2020-03-11T22:31:18.804 INFO:teuthology.task.ansible.out:changed: [smithi155.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2020-03-11T22:31:18.906 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** 2020-03-11T22:31:18.908 INFO:teuthology.task.ansible.out:skipping: no hosts matched 2020-03-11T22:31:18.909 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2020-03-11T22:31:18.911 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** 2020-03-11T22:31:18.912 INFO:teuthology.task.ansible.out:skipping: no hosts matched PLAY RECAP ********************************************************************* smithi155.front.sepia.ceph.com : ok=111 changed=33 unreachable=0 failed=0 skipped=171 rescued=0 ignored=3 Wednesday 11 March 2020 22:31:18 +0000 (0:00:00.300) 0:02:14.479 ******* =============================================================================== 2020-03-11T22:31:18.912 INFO:teuthology.task.ansible.out:common : nrpe - Remove SELinux policy package -------------------------- 16.47s common : nrpe - Load SELinux policy package ---------------------------- 16.34s testnode : Zap all non-root disks --------------------------------------- 7.96s testnode : Install packages --------------------------------------------- 7.90s 2020-03-11T22:31:18.912 INFO:teuthology.task.ansible.out:testnode : Base group workaround for BZ1782899 -------------------------- 5.40s common : Install Kerberos Packages (RedHat) ----------------------------- 3.43s testnode : Ensure ceph dependency packages are not present. ------------- 3.04s Gathering Facts --------------------------------------------------------- 2.84s 2020-03-11T22:31:18.913 INFO:teuthology.task.ansible.out:testnode : Ensure ceph packages are not present. ------------------------ 2.65s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.98s testnode : Create logical volume(s) ------------------------------------- 1.94s testnode : Rebuild rpmdb ------------------------------------------------ 1.88s 2020-03-11T22:31:18.913 INFO:teuthology.task.ansible.out:common : nrpe - Set SELinux boolean nagios_run_sudo true ---------------- 1.64s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 1.56s testnode : Configure local mirrorlists ---------------------------------- 1.48s common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/. --- 1.47s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 1.45s testnode : Install packages via pip ------------------------------------- 1.22s common : Configure local epel mirrorlists ------------------------------- 1.06s 2020-03-11T22:31:18.913 INFO:teuthology.task.ansible.out:testnode : Set the system's h 2020-03-11T22:31:18.914 INFO:teuthology.task.ansible.out:ostname ------------------------------------ 1.06s 2020-03-11T22:31:19.197 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi155.front.sepia.ceph.com')] 2020-03-11T22:31:19.199 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2020-03-11T22:31:19.200 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi155.front.sepia.ceph.com', 'timeout': 60} 2020-03-11T22:31:19.270 INFO:teuthology.orchestra.run.smithi155:> true 2020-03-11T22:31:19.357 INFO:teuthology.run_tasks:Running task clock... 2020-03-11T22:31:19.368 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2020-03-11T22:31:19.368 INFO:teuthology.orchestra.run.smithi155:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2020-03-11T22:31:19.432 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2020-03-11T22:31:19.450 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2020-03-11T22:31:19.496 INFO:teuthology.orchestra.run.smithi155.stderr:sudo: ntpd: command not found 2020-03-11T22:31:19.510 INFO:teuthology.orchestra.run.smithi155.stdout:506 Cannot talk to daemon 2020-03-11T22:31:19.527 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntp.service: Unit ntp.service not found. 2020-03-11T22:31:19.544 INFO:teuthology.orchestra.run.smithi155.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2020-03-11T22:31:19.600 INFO:teuthology.orchestra.run.smithi155.stderr:bash: ntpq: command not found 2020-03-11T22:31:19.602 INFO:teuthology.orchestra.run.smithi155.stdout:210 Number of sources = 4 2020-03-11T22:31:19.602 INFO:teuthology.orchestra.run.smithi155.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2020-03-11T22:31:19.602 INFO:teuthology.orchestra.run.smithi155.stdout:=============================================================================== 2020-03-11T22:31:19.603 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:31:19.603 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:31:19.603 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:31:19.603 INFO:teuthology.orchestra.run.smithi155.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2020-03-11T22:31:19.604 INFO:teuthology.run_tasks:Running task install... 2020-03-11T22:31:19.614 DEBUG:teuthology.task.install:project ceph 2020-03-11T22:31:19.614 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'}} 2020-03-11T22:31:19.614 DEBUG:teuthology.task.install:config {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141'} 2020-03-11T22:31:19.614 INFO:teuthology.task.install:Using flavor: basic 2020-03-11T22:31:19.627 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2020-03-11T22:31:19.627 INFO:teuthology.task.install:extra packages: [] 2020-03-11T22:31:19.627 INFO:teuthology.orchestra.run.smithi155:> sudo lsb_release -is 2020-03-11T22:31:19.681 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS 2020-03-11T22:31:19.682 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:31:19.682 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': '4acc1121e47039a3cdf2978ae458c10d153aa141', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': [], 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': [], 'flavor': 'basic', 'extras': None} 2020-03-11T22:31:19.683 INFO:teuthology.orchestra.run.smithi155:> sudo lsb_release -is 2020-03-11T22:31:19.758 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS 2020-03-11T22:31:19.760 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:31:19.760 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:31:19.970 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/ 2020-03-11T22:31:19.971 INFO:teuthology.task.install.rpm:Package version is 15.1.0-1941.g4acc112 2020-03-11T22:31:20.133 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-zafman-octo-diag/4acc1121e47039a3cdf2978ae458c10d153aa141/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2020-03-11T22:31:20.133 INFO:teuthology.orchestra.run.smithi155:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2020-03-11T22:31:20.178 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2020-03-11T22:31:20.178 INFO:teuthology.orchestra.run.smithi155:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/4acc1121e47039a3cdf2978ae458c10d153aa141/;g' /etc/yum.repos.d/ceph.repo ; fi 2020-03-11T22:31:20.249 INFO:teuthology.orchestra.run.smithi155:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2020-03-11T22:31:20.330 INFO:teuthology.orchestra.run.smithi155:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2020-03-11T22:31:20.396 INFO:teuthology.orchestra.run.smithi155.stdout:check_obsoletes = 1 2020-03-11T22:31:20.397 INFO:teuthology.orchestra.run.smithi155:> sudo yum clean all 2020-03-11T22:31:20.793 INFO:teuthology.orchestra.run.smithi155.stdout:239 files removed 2020-03-11T22:31:20.821 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-radosgw 2020-03-11T22:31:21.522 INFO:teuthology.orchestra.run.smithi155.stdout:ceph packages for x86_64 193 kB/s | 69 kB 00:00 2020-03-11T22:31:21.857 INFO:teuthology.orchestra.run.smithi155.stdout:ceph noarch packages 44 kB/s | 14 kB 00:00 2020-03-11T22:31:22.166 INFO:teuthology.orchestra.run.smithi155.stdout:ceph source packages 4.9 kB/s | 1.4 kB 00:00 2020-03-11T22:31:22.320 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-8 - AppStream 48 MB/s | 6.5 MB 00:00 2020-03-11T22:31:23.452 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-8 - Base 42 MB/s | 5.0 MB 00:00 2020-03-11T22:31:24.221 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS-8 - Extras 33 kB/s | 2.1 kB 00:00 2020-03-11T22:31:24.391 INFO:teuthology.orchestra.run.smithi155.stdout:Copr repo for ceph-el8 owned by ktdreyer 303 kB/s | 38 kB 00:00 2020-03-11T22:31:24.757 INFO:teuthology.orchestra.run.smithi155.stdout:Extra Packages for Enterprise Linux 19 MB/s | 6.0 MB 00:00 2020-03-11T22:31:26.011 INFO:teuthology.orchestra.run.smithi155.stdout:lab-extras 5.5 MB/s | 30 kB 00:00 2020-03-11T22:31:26.975 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:31:26.978 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:31:26.978 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2020-03-11T22:31:26.978 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:31:26.978 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:31:26.978 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 9.8 M 2020-03-11T22:31:26.979 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrading: 2020-03-11T22:31:26.979 INFO:teuthology.orchestra.run.smithi155.stdout: librados2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 3.6 M 2020-03-11T22:31:26.979 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 1.9 M 2020-03-11T22:31:26.979 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2020-03-11T22:31:26.980 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base x86_64 2:15.1.0-1941.g4acc112.el8 ceph 5.5 M 2020-03-11T22:31:26.980 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 M 2020-03-11T22:31:26.980 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux x86_64 2:15.1.0-1941.g4acc112.el8 ceph 25 k 2020-03-11T22:31:26.980 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 507 k 2020-03-11T22:31:26.980 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 377 k 2020-03-11T22:31:26.981 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2 x86_64 2:15.1.0-1941.g4acc112.el8 ceph 6.2 M 2020-03-11T22:31:26.981 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 40 k 2020-03-11T22:31:26.981 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common x86_64 2:15.1.0-1941.g4acc112.el8 ceph 49 k 2020-03-11T22:31:26.981 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs x86_64 2:15.1.0-1941.g4acc112.el8 ceph 162 k 2020-03-11T22:31:26.982 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados x86_64 2:15.1.0-1941.g4acc112.el8 ceph 352 k 2020-03-11T22:31:26.982 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 309 k 2020-03-11T22:31:26.982 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw x86_64 2:15.1.0-1941.g4acc112.el8 ceph 113 k 2020-03-11T22:31:26.982 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2020-03-11T22:31:26.983 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2020-03-11T22:31:26.983 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2020-03-11T22:31:26.983 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq x86_64 0.9.0-1.el8 CentOS-Base 47 k 2020-03-11T22:31:26.983 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs x86_64 2.7-6.el8 epel 315 k 2020-03-11T22:31:26.983 INFO:teuthology.orchestra.run.smithi155.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2020-03-11T22:31:26.984 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2020-03-11T22:31:26.984 INFO:teuthology.orchestra.run.smithi155.stdout: leveldb x86_64 1.21-2.el8 lab-extras 179 k 2020-03-11T22:31:26.984 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:31:26.984 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:31:26.985 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:31:26.985 INFO:teuthology.orchestra.run.smithi155.stdout:Install 21 Packages 2020-03-11T22:31:26.985 INFO:teuthology.orchestra.run.smithi155.stdout:Upgrade 2 Packages 2020-03-11T22:31:26.985 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:31:26.986 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 51 M 2020-03-11T22:31:26.986 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:31:28.299 INFO:teuthology.orchestra.run.smithi155.stdout:(1/23): ceph-radosgw-15.1.0-1941.g4acc112.el8.x 7.4 MB/s | 9.8 MB 00:01 2020-03-11T22:31:28.347 INFO:teuthology.orchestra.run.smithi155.stdout:(2/23): ceph-selinux-15.1.0-1941.g4acc112.el8.x 528 kB/s | 25 kB 00:00 2020-03-11T22:31:28.414 INFO:teuthology.orchestra.run.smithi155.stdout:(3/23): libcephfs2-15.1.0-1941.g4acc112.el8.x86 7.3 MB/s | 507 kB 00:00 2020-03-11T22:31:28.471 INFO:teuthology.orchestra.run.smithi155.stdout:(4/23): ceph-base-15.1.0-1941.g4acc112.el8.x86_ 3.7 MB/s | 5.5 MB 00:01 2020-03-11T22:31:28.475 INFO:teuthology.orchestra.run.smithi155.stdout:(5/23): libradosstriper1-15.1.0-1941.g4acc112.e 6.0 MB/s | 377 kB 00:00 2020-03-11T22:31:28.524 INFO:teuthology.orchestra.run.smithi155.stdout:(6/23): python3-ceph-argparse-15.1.0-1941.g4acc 823 kB/s | 40 kB 00:00 2020-03-11T22:31:28.571 INFO:teuthology.orchestra.run.smithi155.stdout:(7/23): python3-ceph-common-15.1.0-1941.g4acc11 1.0 MB/s | 49 kB 00:00 2020-03-11T22:31:28.623 INFO:teuthology.orchestra.run.smithi155.stdout:(8/23): python3-cephfs-15.1.0-1941.g4acc112.el8 3.1 MB/s | 162 kB 00:00 2020-03-11T22:31:28.685 INFO:teuthology.orchestra.run.smithi155.stdout:(9/23): python3-rados-15.1.0-1941.g4acc112.el8. 5.6 MB/s | 352 kB 00:00 2020-03-11T22:31:28.746 INFO:teuthology.orchestra.run.smithi155.stdout:(10/23): python3-rbd-15.1.0-1941.g4acc112.el8.x 4.9 MB/s | 309 kB 00:00 2020-03-11T22:31:28.800 INFO:teuthology.orchestra.run.smithi155.stdout:(11/23): python3-rgw-15.1.0-1941.g4acc112.el8.x 2.1 MB/s | 113 kB 00:00 2020-03-11T22:31:28.834 INFO:teuthology.orchestra.run.smithi155.stdout:(12/23): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2020-03-11T22:31:28.838 INFO:teuthology.orchestra.run.smithi155.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 68 MB/s | 259 kB 00:00 2020-03-11T22:31:28.839 INFO:teuthology.orchestra.run.smithi155.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 33 MB/s | 44 kB 00:00 2020-03-11T22:31:28.841 INFO:teuthology.orchestra.run.smithi155.stdout:(15/23): librabbitmq-0.9.0-1.el8.x86_64.rpm 36 MB/s | 47 kB 00:00 2020-03-11T22:31:28.954 INFO:teuthology.orchestra.run.smithi155.stdout:(16/23): gperftools-libs-2.7-6.el8.x86_64.rpm 2.7 MB/s | 315 kB 00:00 2020-03-11T22:31:28.974 INFO:teuthology.orchestra.run.smithi155.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 3.0 MB/s | 59 kB 00:00 2020-03-11T22:31:28.993 INFO:teuthology.orchestra.run.smithi155.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 3.8 MB/s | 75 kB 00:00 2020-03-11T22:31:29.000 INFO:teuthology.orchestra.run.smithi155.stdout:(19/23): leveldb-1.21-2.el8.x86_64.rpm 27 MB/s | 179 kB 00:00 2020-03-11T22:31:29.456 INFO:teuthology.orchestra.run.smithi155.stdout:(20/23): librados2-15.1.0-1941.g4acc112.el8.x86 7.9 MB/s | 3.6 MB 00:00 2020-03-11T22:31:29.624 INFO:teuthology.orchestra.run.smithi155.stdout:(21/23): librbd1-15.1.0-1941.g4acc112.el8.x86_6 11 MB/s | 1.9 MB 00:00 2020-03-11T22:31:29.695 INFO:teuthology.orchestra.run.smithi155.stdout:(22/23): librgw2-15.1.0-1941.g4acc112.el8.x86_6 5.1 MB/s | 6.2 MB 00:01 2020-03-11T22:31:30.329 INFO:teuthology.orchestra.run.smithi155.stdout:(23/23): ceph-common-15.1.0-1941.g4acc112.el8.x 6.2 MB/s | 21 MB 00:03 2020-03-11T22:31:30.331 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:31:30.332 INFO:teuthology.orchestra.run.smithi155.stdout:Total 15 MB/s | 51 MB 00:03 2020-03-11T22:31:30.337 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:31:30.381 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:31:30.381 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:31:30.727 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:31:30.728 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:31:31.837 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:31:32.039 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2020-03-11T22:31:32.200 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:31:32.605 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2020-03-11T22:31:32.767 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:31:32.842 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:31:33.084 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:31:33.234 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:31:33.455 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:31:33.662 INFO:teuthology.orchestra.run.smithi155.stdout: Upgrading : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:31:33.735 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:31:33.924 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librabbitmq-0.9.0-1.el8.x86_64 7/25 2020-03-11T22:31:34.067 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:31:34.707 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 8/25 2020-03-11T22:31:34.900 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:31:34.970 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 9/25 2020-03-11T22:31:35.126 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 10/25 2020-03-11T22:31:35.287 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:31:35.521 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:31:35.717 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:31:35.883 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:31:35.953 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 14/25 2020-03-11T22:31:36.154 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 15/25 2020-03-11T22:31:36.383 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : gperftools-libs-2.7-6.el8.x86_64 16/25 2020-03-11T22:31:36.541 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : leveldb-1.21-2.el8.x86_64 17/25 2020-03-11T22:31:36.731 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2020-03-11T22:31:36.884 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 19/25 2020-03-11T22:31:40.480 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:31:40.675 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:31:41.355 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 20/25 2020-03-11T22:31:41.665 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:31:41.755 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 21/25 2020-03-11T22:31:41.882 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:31:58.155 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:31:58.155 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /sys 2020-03-11T22:31:58.155 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /proc 2020-03-11T22:31:58.155 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /dev 2020-03-11T22:31:58.155 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /run 2020-03-11T22:31:58.155 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /mnt 2020-03-11T22:31:58.155 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /var/tmp 2020-03-11T22:31:58.156 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /home 2020-03-11T22:31:58.156 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /tmp 2020-03-11T22:31:58.156 INFO:teuthology.orchestra.run.smithi155.stdout:skipping the directory /dev 2020-03-11T22:31:58.156 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:31:59.239 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:31:59.310 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 23/25 2020-03-11T22:31:59.311 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:31:59.490 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2020-03-11T22:31:59.490 INFO:teuthology.orchestra.run.smithi155.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:32:00.149 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:32:00.150 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 1/25 2020-03-11T22:32:00.150 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2/25 2020-03-11T22:32:00.150 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 3/25 2020-03-11T22:32:00.150 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 4/25 2020-03-11T22:32:00.150 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 5/25 2020-03-11T22:32:00.150 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 6/25 2020-03-11T22:32:00.151 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 7/25 2020-03-11T22:32:00.151 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x 8/25 2020-03-11T22:32:00.151 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86 9/25 2020-03-11T22:32:00.151 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 10/25 2020-03-11T22:32:00.151 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 11/25 2020-03-11T22:32:00.151 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 12/25 2020-03-11T22:32:00.151 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 13/25 2020-03-11T22:32:00.151 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/25 2020-03-11T22:32:00.152 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2020-03-11T22:32:00.152 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2020-03-11T22:32:00.152 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librabbitmq-0.9.0-1.el8.x86_64 17/25 2020-03-11T22:32:00.152 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : gperftools-libs-2.7-6.el8.x86_64 18/25 2020-03-11T22:32:00.152 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2020-03-11T22:32:00.152 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2020-03-11T22:32:00.152 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : leveldb-1.21-2.el8.x86_64 21/25 2020-03-11T22:32:00.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-2:15.1.0-1941.g4acc112.el8.x86_64 22/25 2020-03-11T22:32:00.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2020-03-11T22:32:00.153 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 24/25 2020-03-11T22:32:00.938 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2020-03-11T22:32:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:00.939 INFO:teuthology.orchestra.run.smithi155.stdout:Upgraded: 2020-03-11T22:32:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: librados2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.939 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:00.939 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:32:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-radosgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-selinux-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: libradosstriper1-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: librgw2-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.940 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-argparse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: python3-ceph-common-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: librdkafka-0.11.4-1.el8.x86_64 2020-03-11T22:32:00.941 INFO:teuthology.orchestra.run.smithi155.stdout: lttng-ust-2.8.1-11.el8.x86_64 2020-03-11T22:32:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: python3-prettytable-0.7.2-14.el8.noarch 2020-03-11T22:32:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: librabbitmq-0.9.0-1.el8.x86_64 2020-03-11T22:32:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: gperftools-libs-2.7-6.el8.x86_64 2020-03-11T22:32:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: liboath-2.6.2-3.el8.x86_64 2020-03-11T22:32:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: libunwind-1.3.1-3.el8.x86_64 2020-03-11T22:32:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: leveldb-1.21-2.el8.x86_64 2020-03-11T22:32:00.942 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:00.943 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:32:01.284 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-test 2020-03-11T22:32:01.693 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:35 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:32:02.047 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:32:02.048 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:02.048 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2020-03-11T22:32:02.048 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:02.049 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:32:02.049 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test x86_64 2:15.1.0-1941.g4acc112.el8 ceph 42 M 2020-03-11T22:32:02.049 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2020-03-11T22:32:02.049 INFO:teuthology.orchestra.run.smithi155.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2020-03-11T22:32:02.049 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma x86_64 6.8.2-1.el8 CentOS-AppStream 188 k 2020-03-11T22:32:02.049 INFO:teuthology.orchestra.run.smithi155.stdout: socat x86_64 1.7.3.2-6.el8 CentOS-AppStream 298 k 2020-03-11T22:32:02.049 INFO:teuthology.orchestra.run.smithi155.stdout: libxslt x86_64 1.1.32-3.el8 CentOS-Base 249 k 2020-03-11T22:32:02.050 INFO:teuthology.orchestra.run.smithi155.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2020-03-11T22:32:02.050 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:02.050 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:32:02.050 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:02.050 INFO:teuthology.orchestra.run.smithi155.stdout:Install 6 Packages 2020-03-11T22:32:02.050 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:02.051 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 43 M 2020-03-11T22:32:02.051 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 184 M 2020-03-11T22:32:02.051 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:32:02.098 INFO:teuthology.orchestra.run.smithi155.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 3.3 MB/s | 161 kB 00:00 2020-03-11T22:32:02.100 INFO:teuthology.orchestra.run.smithi155.stdout:(2/6): oniguruma-6.8.2-1.el8.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2020-03-11T22:32:02.106 INFO:teuthology.orchestra.run.smithi155.stdout:(3/6): socat-1.7.3.2-6.el8.x86_64.rpm 41 MB/s | 298 kB 00:00 2020-03-11T22:32:02.107 INFO:teuthology.orchestra.run.smithi155.stdout:(4/6): libxslt-1.1.32-3.el8.x86_64.rpm 39 MB/s | 249 kB 00:00 2020-03-11T22:32:02.182 INFO:teuthology.orchestra.run.smithi155.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 913 kB/s | 69 kB 00:00 2020-03-11T22:32:04.189 INFO:teuthology.orchestra.run.smithi155.stdout:(6/6): ceph-test-15.1.0-1941.g4acc112.el8.x86_6 20 MB/s | 42 MB 00:02 2020-03-11T22:32:04.193 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:04.194 INFO:teuthology.orchestra.run.smithi155.stdout:Total 20 MB/s | 43 MB 00:02 2020-03-11T22:32:04.198 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:32:04.220 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:32:04.221 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:32:04.733 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:32:04.733 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:32:05.385 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:32:05.570 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libxslt-1.1.32-3.el8.x86_64 1/6 2020-03-11T22:32:05.786 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2020-03-11T22:32:06.008 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : socat-1.7.3.2-6.el8.x86_64 3/6 2020-03-11T22:32:06.170 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:32:06.272 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: oniguruma-6.8.2-1.el8.x86_64 4/6 2020-03-11T22:32:10.465 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2020-03-11T22:32:10.917 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:32:11.119 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 6/6 2020-03-11T22:32:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 1/6 2020-03-11T22:32:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2020-03-11T22:32:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : oniguruma-6.8.2-1.el8.x86_64 3/6 2020-03-11T22:32:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : socat-1.7.3.2-6.el8.x86_64 4/6 2020-03-11T22:32:11.120 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libxslt-1.1.32-3.el8.x86_64 5/6 2020-03-11T22:32:11.437 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2020-03-11T22:32:11.437 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:11.437 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:32:11.438 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-test-2:15.1.0-1941.g4acc112.el8.x86_64 jq-1.5-12.el8.x86_64 2020-03-11T22:32:11.438 INFO:teuthology.orchestra.run.smithi155.stdout: oniguruma-6.8.2-1.el8.x86_64 socat-1.7.3.2-6.el8.x86_64 2020-03-11T22:32:11.438 INFO:teuthology.orchestra.run.smithi155.stdout: libxslt-1.1.32-3.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2020-03-11T22:32:11.438 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:11.438 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:32:11.690 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph 2020-03-11T22:32:12.089 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:00:46 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:32:12.492 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:32:12.492 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: Package ArchVersion Repository Size 2020-03-11T22:32:12.493 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:12.493 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:32:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: ceph x86_642:15.1.0-1941.g4acc112.el8 ceph 6.4 k 2020-03-11T22:32:12.493 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2020-03-11T22:32:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds x86_642:15.1.0-1941.g4acc112.el8 ceph 2.0 M 2020-03-11T22:32:12.493 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr x86_642:15.1.0-1941.g4acc112.el8 ceph 1.4 M 2020-03-11T22:32:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon x86_642:15.1.0-1941.g4acc112.el8 ceph 4.4 M 2020-03-11T22:32:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd x86_642:15.1.0-1941.g4acc112.el8 ceph 17 M 2020-03-11T22:32:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 165 k 2020-03-11T22:32:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-lxml x86_644.2.3-1.el8 CentOS-AppStream 1.5 M 2020-03-11T22:32:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako noarch1.0.6-13.el8 CentOS-AppStream 157 k 2020-03-11T22:32:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe x86_640.23-19.el8 CentOS-AppStream 39 k 2020-03-11T22:32:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz noarch2017.2-9.el8 CentOS-AppStream 54 k 2020-03-11T22:32:12.494 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webencodings noarch0.5.1-6.el8 CentOS-AppStream 27 k 2020-03-11T22:32:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug noarch0.12.2-4.el8 CentOS-AppStream 457 k 2020-03-11T22:32:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-chardet noarch3.0.4-7.el8 CentOS-Base 195 k 2020-03-11T22:32:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pysocks noarch1.6.8-3.el8 CentOS-Base 34 k 2020-03-11T22:32:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests noarch2.20.0-2.1.el8_1 CentOS-Base 123 k 2020-03-11T22:32:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3 noarch1.24.2-2.el8 CentOS-Base 176 k 2020-03-11T22:32:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot noarch8.2.1-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 145 k 2020-03-11T22:32:12.495 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy noarch18.4.0-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 385 k 2020-03-11T22:32:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco noarch6.2-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 11 k 2020-03-11T22:32:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools noarch2.0-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 18 k 2020-03-11T22:32:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils noarch0.3.5-11.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 49 k 2020-03-11T22:32:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools noarch7.2.0-3.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 59 k 2020-03-11T22:32:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan noarch1.3.2-9.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 283 k 2020-03-11T22:32:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend noarch2.6-1.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 16 k 2020-03-11T22:32:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric noarch0.8.1-17.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 19 k 2020-03-11T22:32:12.496 INFO:teuthology.orchestra.run.smithi155.stdout: python3-singledispatch noarch3.4.0.3-18.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 24 k 2020-03-11T22:32:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora noarch1.14.1-5.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 30 k 2020-03-11T22:32:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: python3-trustme noarch0.5.2-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 27 k 2020-03-11T22:32:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile noarch2.0-2.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 23 k 2020-03-11T22:32:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt x86_643.1.6-2.el8.1 epel 44 k 2020-03-11T22:32:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: python3-beautifulsoup4 noarch4.6.3-2.el8.1 epel 185 k 2020-03-11T22:32:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: python3-waitress noarch1.2.1-2.el8.1 epel 241 k 2020-03-11T22:32:12.497 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob noarch1.8.5-1.el8.1 epel 251 k 2020-03-11T22:32:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webtest noarch2.0.33-1.el8 epel 85 k 2020-03-11T22:32:12.498 INFO:teuthology.orchestra.run.smithi155.stdout:Installing weak dependencies: 2020-03-11T22:32:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: python3-html5lib noarch1:0.999999999-6.el8 CentOS-AppStream 214 k 2020-03-11T22:32:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cssselect noarch0.9.2-10.el8 epel 40 k 2020-03-11T22:32:12.498 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:12.498 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:32:12.498 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:12.499 INFO:teuthology.orchestra.run.smithi155.stdout:Install 36 Packages 2020-03-11T22:32:12.499 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:12.499 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 29 M 2020-03-11T22:32:12.499 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 105 M 2020-03-11T22:32:12.499 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:32:12.795 INFO:teuthology.orchestra.run.smithi155.stdout:(1/36): ceph-15.1.0-1941.g4acc112.el8.x86_64.rp 22 kB/s | 6.4 kB 00:00 2020-03-11T22:32:13.446 INFO:teuthology.orchestra.run.smithi155.stdout:(2/36): ceph-mgr-15.1.0-1941.g4acc112.el8.x86_6 1.5 MB/s | 1.4 MB 00:00 2020-03-11T22:32:13.581 INFO:teuthology.orchestra.run.smithi155.stdout:(3/36): ceph-mds-15.1.0-1941.g4acc112.el8.x86_6 1.9 MB/s | 2.0 MB 00:01 2020-03-11T22:32:13.649 INFO:teuthology.orchestra.run.smithi155.stdout:(4/36): ceph-mgr-modules-core-15.1.0-1941.g4acc 2.4 MB/s | 165 kB 00:00 2020-03-11T22:32:13.683 INFO:teuthology.orchestra.run.smithi155.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 6.1 MB/s | 214 kB 00:00 2020-03-11T22:32:13.702 INFO:teuthology.orchestra.run.smithi155.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 79 MB/s | 1.5 MB 00:00 2020-03-11T22:32:13.705 INFO:teuthology.orchestra.run.smithi155.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 57 MB/s | 157 kB 00:00 2020-03-11T22:32:13.706 INFO:teuthology.orchestra.run.smithi155.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 30 MB/s | 39 kB 00:00 2020-03-11T22:32:13.708 INFO:teuthology.orchestra.run.smithi155.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 37 MB/s | 54 kB 00:00 2020-03-11T22:32:13.709 INFO:teuthology.orchestra.run.smithi155.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 24 MB/s | 27 kB 00:00 2020-03-11T22:32:13.715 INFO:teuthology.orchestra.run.smithi155.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 74 MB/s | 457 kB 00:00 2020-03-11T22:32:13.719 INFO:teuthology.orchestra.run.smithi155.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 62 MB/s | 195 kB 00:00 2020-03-11T22:32:13.720 INFO:teuthology.orchestra.run.smithi155.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 26 MB/s | 34 kB 00:00 2020-03-11T22:32:13.722 INFO:teuthology.orchestra.run.smithi155.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 53 MB/s | 123 kB 00:00 2020-03-11T22:32:13.726 INFO:teuthology.orchestra.run.smithi155.stdout:(15/36): python3-urllib3-1.24.2-2.el8.noarch.rp 59 MB/s | 176 kB 00:00 2020-03-11T22:32:13.790 INFO:teuthology.orchestra.run.smithi155.stdout:(16/36): python3-cheroot-8.2.1-1.el8.noarch.rpm 2.2 MB/s | 145 kB 00:00 2020-03-11T22:32:13.812 INFO:teuthology.orchestra.run.smithi155.stdout:(17/36): python3-cherrypy-18.4.0-1.el8.noarch.r 17 MB/s | 385 kB 00:00 2020-03-11T22:32:13.820 INFO:teuthology.orchestra.run.smithi155.stdout:(18/36): python3-jaraco-6.2-6.el8.noarch.rpm 1.3 MB/s | 11 kB 00:00 2020-03-11T22:32:13.829 INFO:teuthology.orchestra.run.smithi155.stdout:(19/36): python3-jaraco-functools-2.0-4.el8.noa 2.2 MB/s | 18 kB 00:00 2020-03-11T22:32:13.838 INFO:teuthology.orchestra.run.smithi155.stdout:(20/36): python3-logutils-0.3.5-11.el8.noarch.r 5.5 MB/s | 49 kB 00:00 2020-03-11T22:32:13.847 INFO:teuthology.orchestra.run.smithi155.stdout:(21/36): python3-more-itertools-7.2.0-3.el8.noa 6.5 MB/s | 59 kB 00:00 2020-03-11T22:32:13.859 INFO:teuthology.orchestra.run.smithi155.stdout:(22/36): python3-pecan-1.3.2-9.el8.noarch.rpm 22 MB/s | 283 kB 00:00 2020-03-11T22:32:13.867 INFO:teuthology.orchestra.run.smithi155.stdout:(23/36): python3-portend-2.6-1.el8.noarch.rpm 1.9 MB/s | 16 kB 00:00 2020-03-11T22:32:13.876 INFO:teuthology.orchestra.run.smithi155.stdout:(24/36): python3-simplegeneric-0.8.1-17.el8.noa 2.3 MB/s | 19 kB 00:00 2020-03-11T22:32:13.886 INFO:teuthology.orchestra.run.smithi155.stdout:(25/36): python3-singledispatch-3.4.0.3-18.el8. 2.3 MB/s | 24 kB 00:00 2020-03-11T22:32:13.895 INFO:teuthology.orchestra.run.smithi155.stdout:(26/36): python3-tempora-1.14.1-5.el8.noarch.rp 3.3 MB/s | 30 kB 00:00 2020-03-11T22:32:13.904 INFO:teuthology.orchestra.run.smithi155.stdout:(27/36): python3-trustme-0.5.2-4.el8.noarch.rpm 3.1 MB/s | 27 kB 00:00 2020-03-11T22:32:13.912 INFO:teuthology.orchestra.run.smithi155.stdout:(28/36): python3-zc-lockfile-2.0-2.el8.noarch.r 2.8 MB/s | 23 kB 00:00 2020-03-11T22:32:13.988 INFO:teuthology.orchestra.run.smithi155.stdout:(29/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 575 kB/s | 44 kB 00:00 2020-03-11T22:32:14.045 INFO:teuthology.orchestra.run.smithi155.stdout:(30/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 3.2 MB/s | 185 kB 00:00 2020-03-11T22:32:14.064 INFO:teuthology.orchestra.run.smithi155.stdout:(31/36): python3-cssselect-0.9.2-10.el8.noarch. 2.0 MB/s | 40 kB 00:00 2020-03-11T22:32:14.102 INFO:teuthology.orchestra.run.smithi155.stdout:(32/36): python3-waitress-1.2.1-2.el8.1.noarch. 6.2 MB/s | 241 kB 00:00 2020-03-11T22:32:14.122 INFO:teuthology.orchestra.run.smithi155.stdout:(33/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 12 MB/s | 251 kB 00:00 2020-03-11T22:32:14.142 INFO:teuthology.orchestra.run.smithi155.stdout:(34/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.3 MB/s | 85 kB 00:00 2020-03-11T22:32:14.650 INFO:teuthology.orchestra.run.smithi155.stdout:(35/36): ceph-mon-15.1.0-1941.g4acc112.el8.x86_ 2.3 MB/s | 4.4 MB 00:01 2020-03-11T22:32:15.804 INFO:teuthology.orchestra.run.smithi155.stdout:(36/36): ceph-osd-15.1.0-1941.g4acc112.el8.x86_ 7.1 MB/s | 17 MB 00:02 2020-03-11T22:32:15.806 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:15.806 INFO:teuthology.orchestra.run.smithi155.stdout:Total 8.9 MB/s | 29 MB 00:03 2020-03-11T22:32:15.826 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:32:15.866 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:32:15.866 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:32:16.229 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:32:16.229 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:32:17.713 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:32:17.930 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 1/36 2020-03-11T22:32:18.165 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 2/36 2020-03-11T22:32:18.421 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 3/36 2020-03-11T22:32:18.595 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 4/36 2020-03-11T22:32:18.785 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 5/36 2020-03-11T22:32:18.985 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 6/36 2020-03-11T22:32:19.150 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-trustme-0.5.2-4.el8.noarch 7/36 2020-03-11T22:32:19.323 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 8/36 2020-03-11T22:32:19.488 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 9/36 2020-03-11T22:32:19.663 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 10/36 2020-03-11T22:32:19.808 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 11/36 2020-03-11T22:32:20.033 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 12/36 2020-03-11T22:32:20.276 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cheroot-8.2.1-1.el8.noarch 13/36 2020-03-11T22:32:20.481 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2020-03-11T22:32:20.754 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-urllib3-1.24.2-2.el8.noarch 15/36 2020-03-11T22:32:21.016 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2020-03-11T22:32:21.264 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2020-03-11T22:32:21.470 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2020-03-11T22:32:21.722 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2020-03-11T22:32:21.980 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2020-03-11T22:32:22.339 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2020-03-11T22:32:22.624 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2020-03-11T22:32:22.848 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2020-03-11T22:32:23.054 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2020-03-11T22:32:23.240 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2020-03-11T22:32:23.467 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2020-03-11T22:32:23.697 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2020-03-11T22:32:23.954 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2020-03-11T22:32:24.310 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2020-03-11T22:32:24.600 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2020-03-11T22:32:24.983 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 31/36 2020-03-11T22:32:25.166 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:32:26.578 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 32/36 2020-03-11T22:32:26.767 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:32:27.166 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 33/36 2020-03-11T22:32:27.334 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:32:27.527 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 34/36 2020-03-11T22:32:27.717 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:32:27.736 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 35/36 2020-03-11T22:32:27.840 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:32:28.180 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 36/36 2020-03-11T22:32:28.180 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-2:15.1.0-1941.g4acc112.el8.x86_64 1/36 2020-03-11T22:32:28.180 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2/36 2020-03-11T22:32:28.180 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 3/36 2020-03-11T22:32:28.180 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 4/36 2020-03-11T22:32:28.181 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 5/36 2020-03-11T22:32:28.181 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.n 6/36 2020-03-11T22:32:28.181 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2020-03-11T22:32:28.181 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2020-03-11T22:32:28.181 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2020-03-11T22:32:28.181 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2020-03-11T22:32:28.181 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2020-03-11T22:32:28.182 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2020-03-11T22:32:28.182 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2020-03-11T22:32:28.182 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2020-03-11T22:32:28.182 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2020-03-11T22:32:28.182 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2020-03-11T22:32:28.182 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-urllib3-1.24.2-2.el8.noarch 17/36 2020-03-11T22:32:28.182 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cheroot-8.2.1-1.el8.noarch 18/36 2020-03-11T22:32:28.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 19/36 2020-03-11T22:32:28.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 20/36 2020-03-11T22:32:28.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 21/36 2020-03-11T22:32:28.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 22/36 2020-03-11T22:32:28.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 23/36 2020-03-11T22:32:28.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 24/36 2020-03-11T22:32:28.183 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-portend-2.6-1.el8.noarch 25/36 2020-03-11T22:32:28.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 26/36 2020-03-11T22:32:28.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 27/36 2020-03-11T22:32:28.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 28/36 2020-03-11T22:32:28.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-trustme-0.5.2-4.el8.noarch 29/36 2020-03-11T22:32:28.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 30/36 2020-03-11T22:32:28.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 31/36 2020-03-11T22:32:28.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 32/36 2020-03-11T22:32:28.184 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 33/36 2020-03-11T22:32:28.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 34/36 2020-03-11T22:32:28.185 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 35/36 2020-03-11T22:32:28.919 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 36/36 2020-03-11T22:32:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:28.920 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:32:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2020-03-11T22:32:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cssselect-0.9.2-10.el8.noarch 2020-03-11T22:32:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mds-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:28.920 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mon-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-osd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-modules-core-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:32:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: python3-lxml-4.2.3-1.el8.x86_64 2020-03-11T22:32:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: python3-mako-1.0.6-13.el8.noarch 2020-03-11T22:32:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: python3-markupsafe-0.23-19.el8.x86_64 2020-03-11T22:32:28.921 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pytz-2017.2-9.el8.noarch 2020-03-11T22:32:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webencodings-0.5.1-6.el8.noarch 2020-03-11T22:32:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2020-03-11T22:32:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: python3-chardet-3.0.4-7.el8.noarch 2020-03-11T22:32:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pysocks-1.6.8-3.el8.noarch 2020-03-11T22:32:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2020-03-11T22:32:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: python3-urllib3-1.24.2-2.el8.noarch 2020-03-11T22:32:28.922 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cheroot-8.2.1-1.el8.noarch 2020-03-11T22:32:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2020-03-11T22:32:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-6.2-6.el8.noarch 2020-03-11T22:32:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2020-03-11T22:32:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: python3-logutils-0.3.5-11.el8.noarch 2020-03-11T22:32:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2020-03-11T22:32:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pecan-1.3.2-9.el8.noarch 2020-03-11T22:32:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: python3-portend-2.6-1.el8.noarch 2020-03-11T22:32:28.923 INFO:teuthology.orchestra.run.smithi155.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2020-03-11T22:32:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2020-03-11T22:32:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: python3-tempora-1.14.1-5.el8.noarch 2020-03-11T22:32:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: python3-trustme-0.5.2-4.el8.noarch 2020-03-11T22:32:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2020-03-11T22:32:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2020-03-11T22:32:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2020-03-11T22:32:28.924 INFO:teuthology.orchestra.run.smithi155.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2020-03-11T22:32:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webob-1.8.5-1.el8.1.noarch 2020-03-11T22:32:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: python3-webtest-2.0.33-1.el8.noarch 2020-03-11T22:32:28.925 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:28.925 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:32:29.282 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-base 2020-03-11T22:32:29.687 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:03 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:32:29.929 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-base-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:32:30.035 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:32:30.036 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2020-03-11T22:32:30.037 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:32:30.220 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install cephadm 2020-03-11T22:32:30.625 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:04 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:32:30.994 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:32:30.994 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:30.995 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2020-03-11T22:32:30.995 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:30.995 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:32:30.995 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm x86_64 2:15.1.0-1941.g4acc112.el8 ceph 36 k 2020-03-11T22:32:30.995 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:30.995 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:32:30.995 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:30.996 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2020-03-11T22:32:30.996 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:30.996 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 36 k 2020-03-11T22:32:30.996 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 129 k 2020-03-11T22:32:30.996 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:32:31.183 INFO:teuthology.orchestra.run.smithi155.stdout:cephadm-15.1.0-1941.g4acc112.el8.x86_64.rpm 190 kB/s | 36 kB 00:00 2020-03-11T22:32:31.184 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:31.184 INFO:teuthology.orchestra.run.smithi155.stdout:Total 189 kB/s | 36 kB 00:00 2020-03-11T22:32:31.186 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:32:31.193 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:32:31.194 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:32:31.195 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:32:31.213 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:32:31.242 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:32:32.928 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:33.200 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:33.408 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:32:33.408 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:33.408 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:32:33.408 INFO:teuthology.orchestra.run.smithi155.stdout: cephadm-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:33.408 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:33.408 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:32:33.562 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-immutable-object-cache 2020-03-11T22:32:33.965 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:07 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:32:34.351 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:32:34.352 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:34.352 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2020-03-11T22:32:34.353 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:34.353 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:32:34.353 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache x86_64 2:15.1.0-1941.g4acc112.el8 ceph 203 k 2020-03-11T22:32:34.353 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:34.353 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:32:34.353 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:34.353 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2020-03-11T22:32:34.353 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:34.354 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 203 k 2020-03-11T22:32:34.354 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 743 k 2020-03-11T22:32:34.354 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:32:34.611 INFO:teuthology.orchestra.run.smithi155.stdout:ceph-immutable-object-cache-15.1.0-1941.g4acc11 789 kB/s | 203 kB 00:00 2020-03-11T22:32:34.611 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:34.611 INFO:teuthology.orchestra.run.smithi155.stdout:Total 785 kB/s | 203 kB 00:00 2020-03-11T22:32:34.613 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:32:34.626 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:32:34.626 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:32:34.672 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:32:34.673 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:32:34.782 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:32:34.941 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:32:35.248 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:32:35.516 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.e 1/1 2020-03-11T22:32:35.516 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:35.516 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:32:35.516 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-immutable-object-cache-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:32:35.517 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:35.517 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:32:35.748 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr 2020-03-11T22:32:36.150 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:10 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:32:36.437 INFO:teuthology.orchestra.run.smithi155.stdout:Package ceph-mgr-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:32:36.547 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:32:36.548 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2020-03-11T22:32:36.548 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:32:36.702 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-dashboard 2020-03-11T22:32:37.098 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:11 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:32:37.478 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:32:37.481 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:37.481 INFO:teuthology.orchestra.run.smithi155.stdout: Package ArchVersion Repository Size 2020-03-11T22:32:37.481 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:37.482 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:32:37.482 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 3.9 M 2020-03-11T22:32:37.482 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2020-03-11T22:32:37.482 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 22 k 2020-03-11T22:32:37.482 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt noarch1.6.1-2.el8 CentOS-Base 43 k 2020-03-11T22:32:37.482 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru noarch0.7-6.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 34 k 2020-03-11T22:32:37.482 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes noarch2.4.1-12.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 197 k 2020-03-11T22:32:37.483 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:37.483 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:32:37.483 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:37.483 INFO:teuthology.orchestra.run.smithi155.stdout:Install 5 Packages 2020-03-11T22:32:37.483 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:37.483 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 4.2 M 2020-03-11T22:32:37.483 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 19 M 2020-03-11T22:32:37.483 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:32:37.523 INFO:teuthology.orchestra.run.smithi155.stdout:(1/5): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2020-03-11T22:32:37.624 INFO:teuthology.orchestra.run.smithi155.stdout:(2/5): python3-repoze-lru-0.7-6.el8.noarch.rpm 373 kB/s | 34 kB 00:00 2020-03-11T22:32:37.676 INFO:teuthology.orchestra.run.smithi155.stdout:(3/5): python3-routes-2.4.1-12.el8.noarch.rpm 3.7 MB/s | 197 kB 00:00 2020-03-11T22:32:37.738 INFO:teuthology.orchestra.run.smithi155.stdout:(4/5): ceph-grafana-dashboards-15.1.0-1941.g4ac 91 kB/s | 22 kB 00:00 2020-03-11T22:32:37.992 INFO:teuthology.orchestra.run.smithi155.stdout:(5/5): ceph-mgr-dashboard-15.1.0-1941.g4acc112. 7.6 MB/s | 3.9 MB 00:00 2020-03-11T22:32:37.995 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:37.995 INFO:teuthology.orchestra.run.smithi155.stdout:Total 8.1 MB/s | 4.2 MB 00:00 2020-03-11T22:32:38.000 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:32:38.009 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:32:38.009 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:32:38.148 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:32:38.149 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:32:38.687 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:32:38.956 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/5 2020-03-11T22:32:39.225 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/5 2020-03-11T22:32:39.430 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:32:40.408 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 4/5 2020-03-11T22:32:40.698 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:32:40.911 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 5/5 2020-03-11T22:32:40.911 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.n 1/5 2020-03-11T22:32:40.911 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2/5 2020-03-11T22:32:40.912 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 3/5 2020-03-11T22:32:40.912 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 4/5 2020-03-11T22:32:41.190 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 5/5 2020-03-11T22:32:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:41.191 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:32:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-dashboard-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:32:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-grafana-dashboards-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:32:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jwt-1.6.1-2.el8.noarch 2020-03-11T22:32:41.191 INFO:teuthology.orchestra.run.smithi155.stdout: python3-repoze-lru-0.7-6.el8.noarch 2020-03-11T22:32:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: python3-routes-2.4.1-12.el8.noarch 2020-03-11T22:32:41.192 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:41.192 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:32:41.379 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-diskprediction-local 2020-03-11T22:32:41.777 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:15 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:32:42.146 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:32:42.148 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:42.148 INFO:teuthology.orchestra.run.smithi155.stdout: Package ArchVersion Repository Size 2020-03-11T22:32:42.148 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:42.148 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:32:42.148 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 8.6 M 2020-03-11T22:32:42.148 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2020-03-11T22:32:42.149 INFO:teuthology.orchestra.run.smithi155.stdout: openblas x86_640.3.3-2.el8 CentOS-AppStream 4.3 M 2020-03-11T22:32:42.149 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-threads x86_640.3.3-2.el8 CentOS-AppStream 4.4 M 2020-03-11T22:32:42.149 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy x86_641:1.14.3-9.el8 CentOS-AppStream 3.7 M 2020-03-11T22:32:42.149 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py x86_641:1.14.3-9.el8 CentOS-AppStream 225 k 2020-03-11T22:32:42.149 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy x86_641.0.0-20.module_el8.1.0+245+c39af44f CentOS-AppStream 15 M 2020-03-11T22:32:42.149 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran x86_648.3.1-4.5.el8 CentOS-Base 638 k 2020-03-11T22:32:42.149 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath x86_648.3.1-4.5.el8 CentOS-Base 169 k 2020-03-11T22:32:42.150 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:42.150 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:32:42.150 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:42.150 INFO:teuthology.orchestra.run.smithi155.stdout:Install 8 Packages 2020-03-11T22:32:42.150 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:42.151 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 37 M 2020-03-11T22:32:42.151 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 205 M 2020-03-11T22:32:42.151 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:32:42.259 INFO:teuthology.orchestra.run.smithi155.stdout:(1/8): openblas-0.3.3-2.el8.x86_64.rpm 40 MB/s | 4.3 MB 00:00 2020-03-11T22:32:42.299 INFO:teuthology.orchestra.run.smithi155.stdout:(2/8): openblas-threads-0.3.3-2.el8.x86_64.rpm 29 MB/s | 4.4 MB 00:00 2020-03-11T22:32:42.307 INFO:teuthology.orchestra.run.smithi155.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 29 MB/s | 225 kB 00:00 2020-03-11T22:32:42.332 INFO:teuthology.orchestra.run.smithi155.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 51 MB/s | 3.7 MB 00:00 2020-03-11T22:32:42.346 INFO:teuthology.orchestra.run.smithi155.stdout:(5/8): libgfortran-8.3.1-4.5.el8.x86_64.rpm 47 MB/s | 638 kB 00:00 2020-03-11T22:32:42.351 INFO:teuthology.orchestra.run.smithi155.stdout:(6/8): libquadmath-8.3.1-4.5.el8.x86_64.rpm 32 MB/s | 169 kB 00:00 2020-03-11T22:32:42.564 INFO:teuthology.orchestra.run.smithi155.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.1.0+24 57 MB/s | 15 MB 00:00 2020-03-11T22:32:44.211 INFO:teuthology.orchestra.run.smithi155.stdout:(8/8): ceph-mgr-diskprediction-local-15.1.0-194 4.2 MB/s | 8.6 MB 00:02 2020-03-11T22:32:44.214 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:44.214 INFO:teuthology.orchestra.run.smithi155.stdout:Total 18 MB/s | 37 MB 00:02 2020-03-11T22:32:44.226 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:32:44.246 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:32:44.246 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:32:44.536 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:32:44.537 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:32:45.080 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:32:45.247 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:32:45.377 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libquadmath-8.3.1-4.5.el8.x86_64 1/8 2020-03-11T22:32:45.514 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:32:46.076 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libgfortran-8.3.1-4.5.el8.x86_64 2/8 2020-03-11T22:32:46.264 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:32:46.806 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: openblas-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:32:46.972 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:32:47.674 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: openblas-threads-0.3.3-2.el8.x86_64 4/8 2020-03-11T22:32:47.992 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:32:48.214 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:32:50.174 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2020-03-11T22:32:51.472 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 7/8 2020-03-11T22:32:51.671 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:32:51.909 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 8/8 2020-03-11T22:32:51.909 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112 1/8 2020-03-11T22:32:51.910 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-0.3.3-2.el8.x86_64 2/8 2020-03-11T22:32:51.910 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : openblas-threads-0.3.3-2.el8.x86_64 3/8 2020-03-11T22:32:51.910 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2020-03-11T22:32:51.910 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2020-03-11T22:32:51.910 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x 6/8 2020-03-11T22:32:51.910 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libgfortran-8.3.1-4.5.el8.x86_64 7/8 2020-03-11T22:32:52.212 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libquadmath-8.3.1-4.5.el8.x86_64 8/8 2020-03-11T22:32:52.212 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:52.212 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:32:52.212 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-diskprediction-local-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:32:52.213 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-0.3.3-2.el8.x86_64 2020-03-11T22:32:52.213 INFO:teuthology.orchestra.run.smithi155.stdout: openblas-threads-0.3.3-2.el8.x86_64 2020-03-11T22:32:52.213 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2020-03-11T22:32:52.213 INFO:teuthology.orchestra.run.smithi155.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2020-03-11T22:32:52.213 INFO:teuthology.orchestra.run.smithi155.stdout: python3-scipy-1.0.0-20.module_el8.1.0+245+c39af44f.x86_64 2020-03-11T22:32:52.213 INFO:teuthology.orchestra.run.smithi155.stdout: libgfortran-8.3.1-4.5.el8.x86_64 2020-03-11T22:32:52.213 INFO:teuthology.orchestra.run.smithi155.stdout: libquadmath-8.3.1-4.5.el8.x86_64 2020-03-11T22:32:52.214 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:52.214 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:32:52.368 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-rook 2020-03-11T22:32:52.768 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:26 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:32:53.127 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:32:53.129 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:53.129 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2020-03-11T22:32:53.129 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:53.129 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:32:53.129 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook noarch 2:15.1.0-1941.g4acc112.el8 2020-03-11T22:32:53.129 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-noarch 24 k 2020-03-11T22:32:53.130 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2020-03-11T22:32:53.130 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2020-03-11T22:32:53.130 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2020-03-11T22:32:53.130 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2020-03-11T22:32:53.130 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2020-03-11T22:32:53.130 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2020-03-11T22:32:53.131 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2020-03-11T22:32:53.131 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2020-03-11T22:32:53.131 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2020-03-11T22:32:53.131 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2020-03-11T22:32:53.131 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes noarch 10.0.1-2.el8 epel 1.1 M 2020-03-11T22:32:53.131 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa noarch 3.4.2-12.el8 epel 73 k 2020-03-11T22:32:53.131 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2020-03-11T22:32:53.131 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:53.132 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:32:53.132 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:53.132 INFO:teuthology.orchestra.run.smithi155.stdout:Install 13 Packages 2020-03-11T22:32:53.132 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:53.133 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 1.9 M 2020-03-11T22:32:53.133 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 23 M 2020-03-11T22:32:53.133 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:32:53.162 INFO:teuthology.orchestra.run.smithi155.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 890 kB/s | 27 kB 00:00 2020-03-11T22:32:53.169 INFO:teuthology.orchestra.run.smithi155.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 17 MB/s | 126 kB 00:00 2020-03-11T22:32:53.170 INFO:teuthology.orchestra.run.smithi155.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 524 kB/s | 20 kB 00:00 2020-03-11T22:32:53.172 INFO:teuthology.orchestra.run.smithi155.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 40 MB/s | 110 kB 00:00 2020-03-11T22:32:53.174 INFO:teuthology.orchestra.run.smithi155.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 29 MB/s | 43 kB 00:00 2020-03-11T22:32:53.174 INFO:teuthology.orchestra.run.smithi155.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 37 MB/s | 155 kB 00:00 2020-03-11T22:32:53.221 INFO:teuthology.orchestra.run.smithi155.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 351 kB/s | 16 kB 00:00 2020-03-11T22:32:53.222 INFO:teuthology.orchestra.run.smithi155.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 692 kB/s | 33 kB 00:00 2020-03-11T22:32:53.253 INFO:teuthology.orchestra.run.smithi155.stdout:(9/13): python3-google-auth-1.1.1-10.el8.noarch 2.6 MB/s | 81 kB 00:00 2020-03-11T22:32:53.269 INFO:teuthology.orchestra.run.smithi155.stdout:(10/13): python3-rsa-3.4.2-12.el8.noarch.rpm 4.5 MB/s | 73 kB 00:00 2020-03-11T22:32:53.284 INFO:teuthology.orchestra.run.smithi155.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.8 MB/s | 61 kB 00:00 2020-03-11T22:32:53.296 INFO:teuthology.orchestra.run.smithi155.stdout:(12/13): ceph-mgr-rook-15.1.0-1941.g4acc112.el8 148 kB/s | 24 kB 00:00 2020-03-11T22:32:53.332 INFO:teuthology.orchestra.run.smithi155.stdout:(13/13): python3-kubernetes-10.0.1-2.el8.noarch 10 MB/s | 1.1 MB 00:00 2020-03-11T22:32:53.333 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:53.333 INFO:teuthology.orchestra.run.smithi155.stdout:Total 9.2 MB/s | 1.9 MB 00:00 2020-03-11T22:32:53.341 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:32:53.354 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:32:53.354 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:32:53.426 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:32:53.426 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:32:53.897 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:32:54.166 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2020-03-11T22:32:54.702 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2020-03-11T22:32:54.900 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-rsa-3.4.2-12.el8.noarch 3/13 2020-03-11T22:32:55.137 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2020-03-11T22:32:55.353 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2020-03-11T22:32:55.571 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2020-03-11T22:32:55.873 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2020-03-11T22:32:56.127 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2020-03-11T22:32:57.193 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2020-03-11T22:32:57.452 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-kubernetes-10.0.1-2.el8.noarch 10/13 2020-03-11T22:32:57.728 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2020-03-11T22:32:57.962 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2020-03-11T22:32:58.154 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:32:58.412 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 13/13 2020-03-11T22:32:58.412 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 1/13 2020-03-11T22:32:58.413 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2020-03-11T22:32:58.413 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2020-03-11T22:32:58.413 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2020-03-11T22:32:58.413 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2020-03-11T22:32:58.413 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2020-03-11T22:32:58.413 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2020-03-11T22:32:58.414 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2020-03-11T22:32:58.414 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2020-03-11T22:32:58.414 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2020-03-11T22:32:58.414 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-kubernetes-10.0.1-2.el8.noarch 11/13 2020-03-11T22:32:58.414 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-rsa-3.4.2-12.el8.noarch 12/13 2020-03-11T22:32:58.778 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2020-03-11T22:32:58.778 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:58.778 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:32:58.779 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-rook-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:32:58.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpatch-1.21-2.el8.noarch 2020-03-11T22:32:58.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-jsonpointer-1.10-11.el8.noarch 2020-03-11T22:32:58.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2020-03-11T22:32:58.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2020-03-11T22:32:58.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2020-03-11T22:32:58.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2020-03-11T22:32:58.779 INFO:teuthology.orchestra.run.smithi155.stdout: python3-cachetools-3.1.1-4.el8.noarch 2020-03-11T22:32:58.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-certifi-2018.10.15-7.el8.noarch 2020-03-11T22:32:58.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2020-03-11T22:32:58.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-kubernetes-10.0.1-2.el8.noarch 2020-03-11T22:32:58.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-rsa-3.4.2-12.el8.noarch 2020-03-11T22:32:58.780 INFO:teuthology.orchestra.run.smithi155.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2020-03-11T22:32:58.780 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:58.780 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:32:58.977 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-mgr-cephadm 2020-03-11T22:32:59.373 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:33 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:32:59.748 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:32:59.748 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:59.749 INFO:teuthology.orchestra.run.smithi155.stdout: Package ArchVersion Repository Size 2020-03-11T22:32:59.749 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:59.749 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:32:59.750 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm noarch2:15.1.0-1941.g4acc112.el8 ceph-noarch 36 k 2020-03-11T22:32:59.750 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2020-03-11T22:32:59.750 INFO:teuthology.orchestra.run.smithi155.stdout: python3-execnet noarch1.7.1-1.el8.2 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 356 k 2020-03-11T22:32:59.750 INFO:teuthology.orchestra.run.smithi155.stdout: python3-remoto noarch1.1.4-4.el8 copr:copr.fedorainfracloud.org:ktdreyer:ceph-el8 48 k 2020-03-11T22:32:59.750 INFO:teuthology.orchestra.run.smithi155.stdout: python3-apipkg noarch1.5-6.el8 epel 21 k 2020-03-11T22:32:59.751 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:59.751 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:32:59.751 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:32:59.751 INFO:teuthology.orchestra.run.smithi155.stdout:Install 4 Packages 2020-03-11T22:32:59.752 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:32:59.752 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 461 k 2020-03-11T22:32:59.753 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 1.4 M 2020-03-11T22:32:59.753 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:32:59.801 INFO:teuthology.orchestra.run.smithi155.stdout:(1/4): python3-remoto-1.1.4-4.el8.noarch.rpm 953 kB/s | 48 kB 00:00 2020-03-11T22:32:59.819 INFO:teuthology.orchestra.run.smithi155.stdout:(2/4): python3-execnet-1.7.1-1.el8.2.noarch.rpm 5.0 MB/s | 356 kB 00:00 2020-03-11T22:32:59.858 INFO:teuthology.orchestra.run.smithi155.stdout:(3/4): python3-apipkg-1.5-6.el8.noarch.rpm 373 kB/s | 21 kB 00:00 2020-03-11T22:32:59.995 INFO:teuthology.orchestra.run.smithi155.stdout:(4/4): ceph-mgr-cephadm-15.1.0-1941.g4acc112.el 148 kB/s | 36 kB 00:00 2020-03-11T22:32:59.997 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:32:59.997 INFO:teuthology.orchestra.run.smithi155.stdout:Total 1.8 MB/s | 461 kB 00:00 2020-03-11T22:33:00.001 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:33:00.008 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:33:00.008 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:33:00.124 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:33:00.125 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:33:00.827 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:33:01.092 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/4 2020-03-11T22:33:01.404 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:33:01.601 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:33:01.745 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:33:01.972 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 4/4 2020-03-11T22:33:01.972 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 1/4 2020-03-11T22:33:01.972 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-execnet-1.7.1-1.el8.2.noarch 2/4 2020-03-11T22:33:01.972 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 3/4 2020-03-11T22:33:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/4 2020-03-11T22:33:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:02.204 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:33:02.204 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-mgr-cephadm-2:15.1.0-1941.g4acc112.el8.noarch 2020-03-11T22:33:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: python3-execnet-1.7.1-1.el8.2.noarch 2020-03-11T22:33:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: python3-remoto-1.1.4-4.el8.noarch 2020-03-11T22:33:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: python3-apipkg-1.5-6.el8.noarch 2020-03-11T22:33:02.205 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:02.205 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:02.357 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install ceph-fuse 2020-03-11T22:33:02.760 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:36 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:03.121 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:03.122 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:03.122 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repository Size 2020-03-11T22:33:03.122 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:03.122 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:33:03.122 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 572 k 2020-03-11T22:33:03.122 INFO:teuthology.orchestra.run.smithi155.stdout:Installing dependencies: 2020-03-11T22:33:03.123 INFO:teuthology.orchestra.run.smithi155.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2020-03-11T22:33:03.123 INFO:teuthology.orchestra.run.smithi155.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2020-03-11T22:33:03.123 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:03.123 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:33:03.123 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:03.123 INFO:teuthology.orchestra.run.smithi155.stdout:Install 3 Packages 2020-03-11T22:33:03.123 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:03.124 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 676 k 2020-03-11T22:33:03.124 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 2.1 M 2020-03-11T22:33:03.124 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:33:03.154 INFO:teuthology.orchestra.run.smithi155.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.6 MB/s | 82 kB 00:00 2020-03-11T22:33:03.158 INFO:teuthology.orchestra.run.smithi155.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 621 kB/s | 21 kB 00:00 2020-03-11T22:33:03.456 INFO:teuthology.orchestra.run.smithi155.stdout:(3/3): ceph-fuse-15.1.0-1941.g4acc112.el8.x86_6 1.7 MB/s | 572 kB 00:00 2020-03-11T22:33:03.458 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:03.458 INFO:teuthology.orchestra.run.smithi155.stdout:Total 2.0 MB/s | 676 kB 00:00 2020-03-11T22:33:03.462 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:33:03.482 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:33:03.483 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:33:03.546 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:33:03.547 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:33:03.713 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:33:03.713 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2020-03-11T22:33:03.713 INFO:teuthology.orchestra.run.smithi155.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2020-03-11T22:33:03.713 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:04.160 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:33:04.376 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:33:04.654 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 3/3 2020-03-11T22:33:04.654 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/3 2020-03-11T22:33:04.654 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2020-03-11T22:33:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2020-03-11T22:33:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:04.994 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:33:04.994 INFO:teuthology.orchestra.run.smithi155.stdout: ceph-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2020-03-11T22:33:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: fuse-common-3.2.1-12.el8.x86_64 2020-03-11T22:33:04.995 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:04.995 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:05.316 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install librados-devel 2020-03-11T22:33:05.723 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:39 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:06.098 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:06.099 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:06.099 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2020-03-11T22:33:06.099 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:06.099 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:33:06.100 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:33:06.100 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:06.100 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:33:06.100 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:06.100 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2020-03-11T22:33:06.100 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:06.101 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 141 k 2020-03-11T22:33:06.101 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 487 k 2020-03-11T22:33:06.101 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:33:06.351 INFO:teuthology.orchestra.run.smithi155.stdout:librados-devel-15.1.0-1941.g4acc112.el8.x86_64. 562 kB/s | 141 kB 00:00 2020-03-11T22:33:06.352 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:06.352 INFO:teuthology.orchestra.run.smithi155.stdout:Total 558 kB/s | 141 kB 00:00 2020-03-11T22:33:06.354 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:33:06.361 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:33:06.361 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:33:06.411 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:33:06.411 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:33:06.617 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:33:06.809 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:07.039 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:07.270 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:07.270 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:07.270 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:33:07.270 INFO:teuthology.orchestra.run.smithi155.stdout: librados-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:07.270 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:07.271 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:07.553 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs2 2020-03-11T22:33:07.958 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:41 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:08.258 INFO:teuthology.orchestra.run.smithi155.stdout:Package libcephfs2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:08.366 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:08.367 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2020-03-11T22:33:08.367 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:08.547 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install libcephfs-devel 2020-03-11T22:33:08.953 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:42 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:09.316 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:09.317 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:09.317 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2020-03-11T22:33:09.317 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:09.317 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:33:09.318 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel x86_64 2:15.1.0-1941.g4acc112.el8 ceph 21 k 2020-03-11T22:33:09.318 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:09.318 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:33:09.318 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:09.318 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2020-03-11T22:33:09.318 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:09.319 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 21 k 2020-03-11T22:33:09.319 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 76 k 2020-03-11T22:33:09.319 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:33:09.644 INFO:teuthology.orchestra.run.smithi155.stdout:libcephfs-devel-15.1.0-1941.g4acc112.el8.x86_64 65 kB/s | 21 kB 00:00 2020-03-11T22:33:09.645 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:09.646 INFO:teuthology.orchestra.run.smithi155.stdout:Total 65 kB/s | 21 kB 00:00 2020-03-11T22:33:09.648 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:33:09.652 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:33:09.652 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:33:09.654 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:33:09.654 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:33:09.725 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:33:09.875 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:10.066 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:10.286 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:10.286 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:10.286 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:33:10.287 INFO:teuthology.orchestra.run.smithi155.stdout: libcephfs-devel-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:10.287 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:10.287 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:10.474 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install librados2 2020-03-11T22:33:10.873 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:44 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:11.123 INFO:teuthology.orchestra.run.smithi155.stdout:Package librados2-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:11.230 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:11.231 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2020-03-11T22:33:11.231 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:11.405 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install librbd1 2020-03-11T22:33:11.806 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:45 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:12.298 INFO:teuthology.orchestra.run.smithi155.stdout:Package librbd1-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:12.405 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:12.406 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2020-03-11T22:33:12.407 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:12.565 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rados 2020-03-11T22:33:12.970 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:46 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:13.248 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rados-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:13.354 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:13.355 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2020-03-11T22:33:13.355 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:13.513 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rgw 2020-03-11T22:33:13.914 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:47 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:14.198 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rgw-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:14.303 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:14.304 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2020-03-11T22:33:14.304 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:14.454 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-cephfs 2020-03-11T22:33:14.855 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:48 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:15.148 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-cephfs-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:15.254 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:15.255 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2020-03-11T22:33:15.255 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:15.423 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install python3-rbd 2020-03-11T22:33:15.823 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:49 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:16.106 INFO:teuthology.orchestra.run.smithi155.stdout:Package python3-rbd-2:15.1.0-1941.g4acc112.el8.x86_64 is already installed. 2020-03-11T22:33:16.211 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:16.212 INFO:teuthology.orchestra.run.smithi155.stdout:Nothing to do. 2020-03-11T22:33:16.213 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:16.365 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-fuse 2020-03-11T22:33:16.770 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:50 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:17.171 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:17.172 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:17.172 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2020-03-11T22:33:17.172 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:17.172 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:33:17.173 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse x86_64 2:15.1.0-1941.g4acc112.el8 ceph 72 k 2020-03-11T22:33:17.173 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:17.173 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:33:17.173 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:17.173 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2020-03-11T22:33:17.173 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:17.173 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 72 k 2020-03-11T22:33:17.174 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 230 k 2020-03-11T22:33:17.174 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:33:17.484 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-fuse-15.1.0-1941.g4acc112.el8.x86_64.rpm 232 kB/s | 72 kB 00:00 2020-03-11T22:33:17.485 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:17.485 INFO:teuthology.orchestra.run.smithi155.stdout:Total 231 kB/s | 72 kB 00:00 2020-03-11T22:33:17.487 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:33:17.499 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:33:17.499 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:33:17.550 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:33:17.551 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:33:18.124 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:33:18.315 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:18.549 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:18.801 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:18.801 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:18.802 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:33:18.802 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-fuse-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:18.802 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:18.802 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:19.110 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-mirror 2020-03-11T22:33:19.518 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:01:53 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:19.926 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:19.926 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:19.927 INFO:teuthology.orchestra.run.smithi155.stdout: Package Arch Version Repo Size 2020-03-11T22:33:19.927 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:19.927 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:33:19.927 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror x86_64 2:15.1.0-1941.g4acc112.el8 ceph 2.7 M 2020-03-11T22:33:19.927 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:19.927 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:33:19.927 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:19.928 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2020-03-11T22:33:19.928 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:19.928 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 2.7 M 2020-03-11T22:33:19.928 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 12 M 2020-03-11T22:33:19.929 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:33:23.319 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-mirror-15.1.0-1941.g4acc112.el8.x86_64.rpm 812 kB/s | 2.7 MB 00:03 2020-03-11T22:33:23.320 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:23.320 INFO:teuthology.orchestra.run.smithi155.stdout:Total 811 kB/s | 2.7 MB 00:03 2020-03-11T22:33:23.322 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:33:23.337 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:33:23.337 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:33:23.398 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:33:23.399 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:33:23.768 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:33:24.750 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:25.074 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:25.340 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:25.340 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:25.340 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:33:25.341 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-mirror-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:25.341 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:25.341 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:25.671 INFO:teuthology.orchestra.run.smithi155:> sudo yum -y install rbd-nbd 2020-03-11T22:33:26.079 INFO:teuthology.orchestra.run.smithi155.stdout:Last metadata expiration check: 0:02:00 ago on Wed 11 Mar 2020 10:31:26 PM UTC. 2020-03-11T22:33:26.470 INFO:teuthology.orchestra.run.smithi155.stdout:Dependencies resolved. 2020-03-11T22:33:26.471 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:26.471 INFO:teuthology.orchestra.run.smithi155.stdout: Package Architecture Version Repository Size 2020-03-11T22:33:26.471 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:26.471 INFO:teuthology.orchestra.run.smithi155.stdout:Installing: 2020-03-11T22:33:26.472 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd x86_64 2:15.1.0-1941.g4acc112.el8 ceph 141 k 2020-03-11T22:33:26.472 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:26.472 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction Summary 2020-03-11T22:33:26.472 INFO:teuthology.orchestra.run.smithi155.stdout:================================================================================ 2020-03-11T22:33:26.472 INFO:teuthology.orchestra.run.smithi155.stdout:Install 1 Package 2020-03-11T22:33:26.472 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:26.473 INFO:teuthology.orchestra.run.smithi155.stdout:Total download size: 141 k 2020-03-11T22:33:26.473 INFO:teuthology.orchestra.run.smithi155.stdout:Installed size: 526 k 2020-03-11T22:33:26.473 INFO:teuthology.orchestra.run.smithi155.stdout:Downloading Packages: 2020-03-11T22:33:26.867 INFO:teuthology.orchestra.run.smithi155.stdout:rbd-nbd-15.1.0-1941.g4acc112.el8.x86_64.rpm 357 kB/s | 141 kB 00:00 2020-03-11T22:33:26.867 INFO:teuthology.orchestra.run.smithi155.stdout:-------------------------------------------------------------------------------- 2020-03-11T22:33:26.867 INFO:teuthology.orchestra.run.smithi155.stdout:Total 356 kB/s | 141 kB 00:00 2020-03-11T22:33:26.869 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction check 2020-03-11T22:33:26.880 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction check succeeded. 2020-03-11T22:33:26.881 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction test 2020-03-11T22:33:26.929 INFO:teuthology.orchestra.run.smithi155.stdout:Transaction test succeeded. 2020-03-11T22:33:26.930 INFO:teuthology.orchestra.run.smithi155.stdout:Running transaction 2020-03-11T22:33:27.042 INFO:teuthology.orchestra.run.smithi155.stdout: Preparing : 1/1 2020-03-11T22:33:27.239 INFO:teuthology.orchestra.run.smithi155.stdout: Installing : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:27.449 INFO:teuthology.orchestra.run.smithi155.stdout: Running scriptlet: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: Verifying : rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 1/1 2020-03-11T22:33:27.715 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:27.716 INFO:teuthology.orchestra.run.smithi155.stdout:Installed: 2020-03-11T22:33:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: rbd-nbd-2:15.1.0-1941.g4acc112.el8.x86_64 2020-03-11T22:33:27.716 INFO:teuthology.orchestra.run.smithi155.stdout: 2020-03-11T22:33:27.716 INFO:teuthology.orchestra.run.smithi155.stdout:Complete! 2020-03-11T22:33:28.038 DEBUG:teuthology.parallel:result is None 2020-03-11T22:33:28.038 INFO:teuthology.orchestra.run.smithi155:> sudo lsb_release -is 2020-03-11T22:33:28.075 INFO:teuthology.orchestra.run.smithi155.stdout:CentOS 2020-03-11T22:33:28.076 DEBUG:teuthology.misc:System to be installed: CentOS 2020-03-11T22:33:28.076 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:33:28.301 INFO:teuthology.orchestra.run.smithi155:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2020-03-11T22:33:28.320 INFO:teuthology.orchestra.run.smithi155.stdout:15.1.0-1941.g4acc112.el8 2020-03-11T22:33:28.320 INFO:teuthology.packaging:The installed version of ceph is 15.1.0-1941.g4acc112.el8 2020-03-11T22:33:28.320 INFO:teuthology.task.install:The correct ceph version 15.1.0-1941.g4acc112 is installed. 2020-03-11T22:33:28.321 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2020-03-11T22:33:28.328 INFO:teuthology.orchestra.run.smithi155:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2020-03-11T22:33:28.387 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2020-03-11T22:33:28.390 INFO:teuthology.orchestra.run.smithi155:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2020-03-11T22:33:28.455 INFO:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/daemon-helper 2020-03-11T22:33:28.522 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2020-03-11T22:33:28.523 INFO:teuthology.orchestra.run.smithi155:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2020-03-11T22:33:28.609 INFO:teuthology.orchestra.run.smithi155:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2020-03-11T22:33:28.675 INFO:teuthology.run_tasks:Running task workunit... 2020-03-11T22:33:28.695 INFO:tasks.workunit:Pulling workunits from ref 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:33:28.695 INFO:tasks.workunit:Making a separate scratch dir for every client... 2020-03-11T22:33:28.695 INFO:tasks.workunit:timeout=3h 2020-03-11T22:33:28.695 INFO:tasks.workunit:cleanup=True 2020-03-11T22:33:28.696 INFO:teuthology.orchestra.run.smithi155:> stat -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:33:28.729 INFO:teuthology.orchestra.run.smithi155.stderr:stat: cannot stat '/home/ubuntu/cephtest/mnt.0': No such file or directory 2020-03-11T22:33:28.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2020-03-11T22:33:28.729 INFO:teuthology.orchestra.run.smithi155:> mkdir -- /home/ubuntu/cephtest/mnt.0 2020-03-11T22:33:28.783 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2020-03-11T22:33:28.784 INFO:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2020-03-11T22:33:28.837 INFO:teuthology.orchestra.run.smithi155:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 4acc1121e47039a3cdf2978ae458c10d153aa141 2020-03-11T22:33:28.941 INFO:tasks.workunit.client.0.smithi155.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2020-03-11T22:35:04.810 INFO:tasks.workunit.client.0.smithi155.stderr:Note: checking out '4acc1121e47039a3cdf2978ae458c10d153aa141'. 2020-03-11T22:35:04.811 INFO:tasks.workunit.client.0.smithi155.stderr: 2020-03-11T22:35:04.811 INFO:tasks.workunit.client.0.smithi155.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2020-03-11T22:35:04.811 INFO:tasks.workunit.client.0.smithi155.stderr:changes and commit them, and you can discard any commits you make in this 2020-03-11T22:35:04.811 INFO:tasks.workunit.client.0.smithi155.stderr:state without impacting any branches by performing another checkout. 2020-03-11T22:35:04.811 INFO:tasks.workunit.client.0.smithi155.stderr: 2020-03-11T22:35:04.811 INFO:tasks.workunit.client.0.smithi155.stderr:If you want to create a new branch to retain commits you create, you may 2020-03-11T22:35:04.811 INFO:tasks.workunit.client.0.smithi155.stderr:do so (now or later) by using -b with the checkout command again. Example: 2020-03-11T22:35:04.812 INFO:tasks.workunit.client.0.smithi155.stderr: 2020-03-11T22:35:04.812 INFO:tasks.workunit.client.0.smithi155.stderr: git checkout -b 2020-03-11T22:35:04.812 INFO:tasks.workunit.client.0.smithi155.stderr: 2020-03-11T22:35:04.812 INFO:tasks.workunit.client.0.smithi155.stderr:HEAD is now at 4acc1121e4 Over 30 seconds monitor pg states 2020-03-11T22:35:04.814 INFO:teuthology.orchestra.run.smithi155:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2020-03-11T22:35:04.886 DEBUG:teuthology.orchestra.remote:smithi155:/home/ubuntu/cephtest/workunits.list.client.0 is 1KB 2020-03-11T22:35:04.934 INFO:tasks.workunit:Running workunits matching osd on client.0... 2020-03-11T22:35:04.934 INFO:tasks.workunit:Running workunit osd/divergent-priors.sh... 2020-03-11T22:35:04.934 INFO:teuthology.orchestra.run.smithi155:workunit test osd/divergent-priors.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh 2020-03-11T22:35:04.962 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:35:04.972 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:35:04.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:35:04.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:35:04.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:35:04.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:35:04.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:35:04.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:35:04.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:35:04.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:35:04.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/divergent-priors 2020-03-11T22:35:04.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:21: run: local dir=td/divergent-priors 2020-03-11T22:35:04.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:22: run: shift 2020-03-11T22:35:04.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: export loglen=12 2020-03-11T22:35:04.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:25: run: loglen=12 2020-03-11T22:35:04.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: export divisor=3 2020-03-11T22:35:04.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:26: run: divisor=3 2020-03-11T22:35:04.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: expr 12 / 2 2020-03-11T22:35:04.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: export trim=6 2020-03-11T22:35:04.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:27: run: trim=6 2020-03-11T22:35:04.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: expr 6 / 3 2020-03-11T22:35:04.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: export DIVERGENT_WRITE=2 2020-03-11T22:35:04.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:28: run: DIVERGENT_WRITE=2 2020-03-11T22:35:04.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: expr 6 / 3 2020-03-11T22:35:04.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: export DIVERGENT_REMOVE=2 2020-03-11T22:35:04.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:29: run: DIVERGENT_REMOVE=2 2020-03-11T22:35:04.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: expr 6 / 3 2020-03-11T22:35:04.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: export DIVERGENT_CREATE=2 2020-03-11T22:35:04.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:30: run: DIVERGENT_CREATE=2 2020-03-11T22:35:04.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: export poolname=test 2020-03-11T22:35:04.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:31: run: poolname=test 2020-03-11T22:35:04.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: export testobjects=100 2020-03-11T22:35:04.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:32: run: testobjects=100 2020-03-11T22:35:04.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: export CEPH_MON=127.0.0.1:7115 2020-03-11T22:35:04.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:34: run: CEPH_MON=127.0.0.1:7115 2020-03-11T22:35:04.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:35: run: export CEPH_ARGS 2020-03-11T22:35:04.978 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: uuidgen 2020-03-11T22:35:04.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:36: run: CEPH_ARGS+='--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none ' 2020-03-11T22:35:04.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:37: run: CEPH_ARGS+='--mon-host=127.0.0.1:7115 ' 2020-03-11T22:35:04.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:39: run: CEPH_ARGS+='--osd_force_auth_primary_missing_objects=1000000 ' 2020-03-11T22:35:04.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:40: run: CEPH_ARGS+='--osd_debug_pg_log_writeout=true ' 2020-03-11T22:35:04.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:41: run: CEPH_ARGS+='--osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:04.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: set 2020-03-11T22:35:04.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:35:04.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:43: run: local 'funcs=TEST_divergent 2020-03-11T22:35:04.983 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_divergent_2 2020-03-11T22:35:04.983 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_divergent_3 2020-03-11T22:35:04.983 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_divergent_ec' 2020-03-11T22:35:04.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:35:04.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:35:04.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:35:04.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:35:04.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:35:04.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:35:04.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:35:04.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:35:04.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:35:04.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:35:04.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:35:04.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:35:04.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:35:04.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:35:04.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:35:04.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:35:04.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:35:04.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:35:04.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:35:04.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:35:04.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:35:04.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:35:04.988 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:35:04.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:35:04.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:35:04.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:35:04.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:35:04.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:04.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:04.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17464 2020-03-11T22:35:04.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:35:04.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:35:04.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:35:04.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:35:04.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:04.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:04.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17464 2020-03-11T22:35:04.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:35:04.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:35:04.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:35:04.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:35:04.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:35:04.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent td/divergent-priors 2020-03-11T22:35:04.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:57: TEST_divergent: local dir=td/divergent-priors 2020-03-11T22:35:04.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:60: TEST_divergent: local dummyfile=/etc/fstab 2020-03-11T22:35:04.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:61: TEST_divergent: local dummyfile2=/etc/resolv.conf 2020-03-11T22:35:04.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:63: TEST_divergent: local num_osds=3 2020-03-11T22:35:04.996 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: expr 3 - 1 2020-03-11T22:35:04.996 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: seq 0 2 2020-03-11T22:35:05.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:64: TEST_divergent: local 'osds=0 2020-03-11T22:35:05.006 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:35:05.006 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:35:05.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:65: TEST_divergent: run_mon td/divergent-priors a 2020-03-11T22:35:05.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:35:05.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:35:05.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:35:05.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:35:05.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:35:05.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:35:05.393 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:35:05.393 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:05.393 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:05.393 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:05.394 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:05.394 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:05.394 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:35:05.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:35:05.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:35:05.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:35:05.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:35:05.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:35:05.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:35:05.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:35:05.563 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:35:05.563 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:35:05.563 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:35:05.564 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:05.564 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:05.564 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:05.564 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-mon.a.asok 2020-03-11T22:35:05.564 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:35:05.564 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17464/ceph-mon.a.asok config get fsid 2020-03-11T22:35:05.650 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:35:05.650 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:35:05.650 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:35:05.650 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:35:05.651 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:35:05.651 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:35:05.651 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:35:05.651 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:35:05.651 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:05.651 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:05.651 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:05.652 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-mon.a.asok 2020-03-11T22:35:05.652 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:35:05.652 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17464/ceph-mon.a.asok config get mon_host 2020-03-11T22:35:05.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:66: TEST_divergent: run_mgr td/divergent-priors x 2020-03-11T22:35:05.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:35:05.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:35:05.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:35:05.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:35:05.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:35:05.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:35:06.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:35:06.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:06.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:06.228 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:06.228 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:06.228 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:06.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:35:06.229 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:35:06.229 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:35:06.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:35:06.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 0 2020-03-11T22:35:06.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:35:06.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:35:06.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:35:06.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:35:06.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:35:06.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:06.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:06.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:06.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:06.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:35:06.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:35:06.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:06.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:35:06.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:06.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:35:06.260 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:06.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:06.262 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:06.262 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:06.262 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:06.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:35:06.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:35:06.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:06.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:35:06.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:35:06.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:06.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:06.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:06.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:06.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:06.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:35:06.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:35:06.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:35:06.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:35:06.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e73ac5a4-38f7-49d2-9ebe-aa4155e118b1 2020-03-11T22:35:06.264 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 e73ac5a4-38f7-49d2-9ebe-aa4155e118b1' 2020-03-11T22:35:06.264 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:35:06.264 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 e73ac5a4-38f7-49d2-9ebe-aa4155e118b1 2020-03-11T22:35:06.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCaZ2levM5IEBAAUdHixBt9OeR5MhX60KtmsA== 2020-03-11T22:35:06.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCaZ2levM5IEBAAUdHixBt9OeR5MhX60KtmsA=="}' 2020-03-11T22:35:06.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e73ac5a4-38f7-49d2-9ebe-aa4155e118b1 -i td/divergent-priors/0/new.json 2020-03-11T22:35:06.637 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:35:06.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:35:06.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCaZ2levM5IEBAAUdHixBt9OeR5MhX60KtmsA== --osd-uuid e73ac5a4-38f7-49d2-9ebe-aa4155e118b1 2020-03-11T22:35:06.681 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:06.674+0000 7fa9ee608ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:06.682 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:06.675+0000 7fa9ee608ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:06.682 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:06.677+0000 7fa9ee608ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:06.683 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:06.677+0000 7fa9ee608ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:35:06.683 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:06.677+0000 7fa9ee608ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:35:06.683 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:06.677+0000 7fa9ee608ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:35:06.726 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:06.717+0000 7fa9ee608ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:35:09.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:35:09.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:35:09.734 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2020-03-11T22:35:09.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:35:09.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:35:09.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:35:09.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:35:09.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:09.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:35:09.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:35:09.999 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:35:10.025 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:10.023+0000 7ff50268cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:10.027 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:10.025+0000 7ff50268cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:10.028 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:10.026+0000 7ff50268cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:10.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:35:10.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:10.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:35:10.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:10.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:10.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:10.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:10.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:10.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:35:10.224 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:35:10.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:35:10.546 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:10.544+0000 7ff50268cec0 -1 Falling back to public interface 2020-03-11T22:35:11.246 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:11.244+0000 7ff50268cec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:35:11.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:35:11.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:11.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:35:11.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:11.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:35:11.450 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:35:11.848 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/17845,v1:127.0.0.1:6803/17845] [v2:127.0.0.1:6804/17845,v1:127.0.0.1:6805/17845] exists,up e73ac5a4-38f7-49d2-9ebe-aa4155e118b1 2020-03-11T22:35:11.848 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 748827f2-8dc8-4162-b402-942b80b4efe3 2020-03-11T22:35:11.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:11.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:11.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:11.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:35:11.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 1 2020-03-11T22:35:11.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:35:11.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:35:11.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:35:11.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:35:11.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:35:11.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:11.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:11.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:11.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:11.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:35:11.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:35:11.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:11.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:35:11.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:11.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:35:11.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:11.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:11.852 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:11.852 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:11.852 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:11.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:35:11.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:35:11.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:11.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:35:11.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:35:11.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:11.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:11.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:11.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:11.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:11.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:35:11.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:35:11.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:35:11.854 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:35:11.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=748827f2-8dc8-4162-b402-942b80b4efe3 2020-03-11T22:35:11.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 748827f2-8dc8-4162-b402-942b80b4efe3' 2020-03-11T22:35:11.855 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:35:11.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCfZ2lehEdhMxAAkITx+3UaRULcNhOFiS6V3w== 2020-03-11T22:35:11.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCfZ2lehEdhMxAAkITx+3UaRULcNhOFiS6V3w=="}' 2020-03-11T22:35:11.865 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 748827f2-8dc8-4162-b402-942b80b4efe3 -i td/divergent-priors/1/new.json 2020-03-11T22:35:12.387 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:35:12.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:35:12.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCfZ2lehEdhMxAAkITx+3UaRULcNhOFiS6V3w== --osd-uuid 748827f2-8dc8-4162-b402-942b80b4efe3 2020-03-11T22:35:12.425 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:12.423+0000 7fd22084fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:12.427 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:12.425+0000 7fd22084fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:12.429 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:12.427+0000 7fd22084fec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:12.430 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:12.427+0000 7fd22084fec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:35:12.430 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:12.427+0000 7fd22084fec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:35:12.430 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:12.427+0000 7fd22084fec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:35:12.479 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:12.477+0000 7fd22084fec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:35:15.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:35:15.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:35:15.426 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2020-03-11T22:35:15.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:35:15.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:35:15.878 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:35:15.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:35:15.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:15.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:35:15.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:35:15.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:35:15.905 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:15.902+0000 7fd7fa929ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:15.921 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:15.918+0000 7fd7fa929ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:15.922 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:15.920+0000 7fd7fa929ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:16.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:35:16.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:16.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:35:16.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:16.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:16.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:16.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:16.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:35:16.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:16.228 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:35:16.438 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:16.435+0000 7fd7fa929ec0 -1 Falling back to public interface 2020-03-11T22:35:16.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:35:17.177 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:17.175+0000 7fd7fa929ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:35:17.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:35:17.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:17.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:35:17.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:17.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:35:17.600 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:35:17.957 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/18256,v1:127.0.0.1:6811/18256] [v2:127.0.0.1:6812/18256,v1:127.0.0.1:6813/18256] exists,up 748827f2-8dc8-4162-b402-942b80b4efe3 2020-03-11T22:35:17.958 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 ea275d10-e3f1-4218-8e10-8f2a58b11e5b 2020-03-11T22:35:17.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:17.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:17.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:17.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:67: TEST_divergent: for i in $osds 2020-03-11T22:35:17.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:69: TEST_divergent: run_osd td/divergent-priors 2 2020-03-11T22:35:17.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:35:17.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:35:17.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:35:17.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:35:17.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:35:17.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:35:17.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:35:17.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:35:17.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:35:17.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:35:17.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:35:17.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:35:17.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:35:17.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:35:17.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:35:17.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:35:17.962 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:35:17.962 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:35:17.962 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:17.962 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:17.962 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:35:17.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:35:17.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:35:17.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:35:17.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:35:17.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:35:17.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:35:17.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:35:17.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:35:17.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:17.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:35:17.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:35:17.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:35:17.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:35:17.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=ea275d10-e3f1-4218-8e10-8f2a58b11e5b 2020-03-11T22:35:17.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 ea275d10-e3f1-4218-8e10-8f2a58b11e5b' 2020-03-11T22:35:17.965 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:35:17.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQClZ2leRzIVOhAAV0Y2Pxx6+vqLpfKOR4y9LA== 2020-03-11T22:35:17.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQClZ2leRzIVOhAAV0Y2Pxx6+vqLpfKOR4y9LA=="}' 2020-03-11T22:35:17.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new ea275d10-e3f1-4218-8e10-8f2a58b11e5b -i td/divergent-priors/2/new.json 2020-03-11T22:35:18.516 INFO:tasks.workunit.client.0.smithi155.stdout:2 2020-03-11T22:35:18.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:35:18.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQClZ2leRzIVOhAAV0Y2Pxx6+vqLpfKOR4y9LA== --osd-uuid ea275d10-e3f1-4218-8e10-8f2a58b11e5b 2020-03-11T22:35:18.554 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:18.552+0000 7fb11589bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:18.585 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:18.582+0000 7fb11589bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:18.586 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:18.584+0000 7fb11589bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:18.587 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:18.584+0000 7fb11589bec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:35:18.587 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:18.584+0000 7fb11589bec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:35:18.587 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:18.584+0000 7fb11589bec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:35:18.635 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:18.632+0000 7fb11589bec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:35:21.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:35:21.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:35:21.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:35:21.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:35:21.662 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2020-03-11T22:35:22.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:35:22.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:35:22.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:35:22.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:35:22.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:35:22.123 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:35:22.148 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:22.145+0000 7f67493fcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:22.149 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:22.147+0000 7f67493fcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:22.151 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:22.148+0000 7f67493fcec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:35:22.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:35:22.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:35:22.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:35:22.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:35:22.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:35:22.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:22.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:35:22.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:22.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:35:22.474 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:35:22.667 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:22.664+0000 7f67493fcec0 -1 Falling back to public interface 2020-03-11T22:35:22.837 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:35:23.385 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:35:23.382+0000 7f67493fcec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:35:23.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:35:23.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:35:23.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:35:23.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:35:23.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:35:23.839 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:35:24.208 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/18619,v1:127.0.0.1:6819/18619] [v2:127.0.0.1:6820/18619,v1:127.0.0.1:6821/18619] exists,up ea275d10-e3f1-4218-8e10-8f2a58b11e5b 2020-03-11T22:35:24.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:35:24.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:35:24.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:35:24.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:72: TEST_divergent: ceph osd set noout 2020-03-11T22:35:24.970 INFO:tasks.workunit.client.0.smithi155.stderr:noout is set 2020-03-11T22:35:24.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:73: TEST_divergent: ceph osd set noin 2020-03-11T22:35:25.688 INFO:tasks.workunit.client.0.smithi155.stderr:noin is set 2020-03-11T22:35:25.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:74: TEST_divergent: ceph osd set nodown 2020-03-11T22:35:26.490 INFO:tasks.workunit.client.0.smithi155.stderr:nodown is set 2020-03-11T22:35:26.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:75: TEST_divergent: create_pool test 1 1 2020-03-11T22:35:26.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:35:27.080 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test' already exists 2020-03-11T22:35:27.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:35:28.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:76: TEST_divergent: ceph osd pool set test size 3 2020-03-11T22:35:28.712 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 size to 3 2020-03-11T22:35:28.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:77: TEST_divergent: ceph osd pool set test min_size 2 2020-03-11T22:35:29.455 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 min_size to 2 2020-03-11T22:35:29.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:79: TEST_divergent: flush_pg_stats 2020-03-11T22:35:29.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:35:29.464 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:35:29.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:35:29.810 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:35:29.810 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:35:29.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:35:29.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:29.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:35:29.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:35:29.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:35:29.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:29.940 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:35:30.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672966 2020-03-11T22:35:30.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966' 2020-03-11T22:35:30.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:30.071 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:35:30.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509444 2020-03-11T22:35:30.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672966 2-64424509444' 2020-03-11T22:35:30.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:30.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:35:30.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:30.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:35:30.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:35:30.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:30.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:35:30.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:35:30.204 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:35:30.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:30.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:35:30.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:30.564 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672966 2020-03-11T22:35:30.564 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:30.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:35:30.566 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672966 2020-03-11T22:35:30.566 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:30.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672966 2020-03-11T22:35:30.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672966' 2020-03-11T22:35:30.567 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672966 2020-03-11T22:35:30.568 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:35:30.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672966 -lt 42949672966 2020-03-11T22:35:31.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:31.098 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509444 2020-03-11T22:35:31.098 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:31.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:35:31.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509444 2020-03-11T22:35:31.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:31.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509444 2020-03-11T22:35:31.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509444' 2020-03-11T22:35:31.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:35:31.099 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509444 2020-03-11T22:35:31.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509444 -lt 64424509444 2020-03-11T22:35:31.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:80: TEST_divergent: wait_for_clean 2020-03-11T22:35:31.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:35:31.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:35:31.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:35:31.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:35:31.287 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:35:31.287 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:35:31.288 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:35:31.288 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:35:31.288 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:35:31.288 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:35:31.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:35:31.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:35:31.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:35:31.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:35:31.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:35:31.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:35:31.720 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:35:31.720 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:35:31.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:35:31.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:31.721 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:35:31.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836488 2020-03-11T22:35:31.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836488' 2020-03-11T22:35:31.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:31.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:35:31.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672968 2020-03-11T22:35:31.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836488 1-42949672968' 2020-03-11T22:35:31.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:31.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:35:32.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509446 2020-03-11T22:35:32.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836488 1-42949672968 2-64424509446' 2020-03-11T22:35:32.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:32.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836488 2020-03-11T22:35:32.111 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:32.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:35:32.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836488 2020-03-11T22:35:32.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:32.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836488 2020-03-11T22:35:32.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2020-03-11T22:35:32.114 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836488 2020-03-11T22:35:32.115 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:32.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836488 2020-03-11T22:35:32.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:32.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672968 2020-03-11T22:35:32.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:32.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:35:32.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672968 2020-03-11T22:35:32.460 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:32.460 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672968 2020-03-11T22:35:32.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672968 2020-03-11T22:35:32.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672968' 2020-03-11T22:35:32.461 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:35:32.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672968 -lt 42949672968 2020-03-11T22:35:32.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:32.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509446 2020-03-11T22:35:32.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:32.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:35:32.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509446 2020-03-11T22:35:32.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:32.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509446 2020-03-11T22:35:32.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509446' 2020-03-11T22:35:32.812 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509446 2020-03-11T22:35:32.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:35:33.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509446 -lt 64424509446 2020-03-11T22:35:33.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:35:33.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:33.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:33.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:35:33.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:33.627 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:33.627 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:33.627 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:33.627 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:33.627 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:33.627 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:33.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:35:33.959 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:33.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:33.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:34.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:35:34.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:35:34.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:35:34.405 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: ceph pg dump pgs --format=json 2020-03-11T22:35:34.405 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: jq '.pg_stats[0].up_primary' 2020-03-11T22:35:34.751 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:35:34.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:83: TEST_divergent: local divergent=1 2020-03-11T22:35:34.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:84: TEST_divergent: echo 'primary and soon to be divergent is 1' 2020-03-11T22:35:34.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:85: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:34.760 INFO:tasks.workunit.client.0.smithi155.stdout:primary and soon to be divergent is 1 2020-03-11T22:35:35.081 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:35.081 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:35:27.639709+0000 0'0 27:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:35:35.081 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:35:35.081 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:35.083 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:35:35.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:86: TEST_divergent: local non_divergent= 2020-03-11T22:35:35.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:35:35.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 0 = 1 ']' 2020-03-11T22:35:35.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0' 2020-03-11T22:35:35.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:35:35.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 1 = 1 ']' 2020-03-11T22:35:35.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:90: TEST_divergent: continue 2020-03-11T22:35:35.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:87: TEST_divergent: for i in $osds 2020-03-11T22:35:35.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:89: TEST_divergent: '[' 2 = 1 ']' 2020-03-11T22:35:35.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:92: TEST_divergent: non_divergent=' 0 2' 2020-03-11T22:35:35.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:95: TEST_divergent: echo 'writing initial objects' 2020-03-11T22:35:35.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: seq 1 100 2020-03-11T22:35:35.093 INFO:tasks.workunit.client.0.smithi155.stdout:writing initial objects 2020-03-11T22:35:35.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:35.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_1 /etc/fstab 2020-03-11T22:35:35.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:35.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_2 /etc/fstab 2020-03-11T22:35:35.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:35.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_3 /etc/fstab 2020-03-11T22:35:35.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:35.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_4 /etc/fstab 2020-03-11T22:35:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:36.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_5 /etc/fstab 2020-03-11T22:35:36.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:36.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_6 /etc/fstab 2020-03-11T22:35:36.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:36.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_7 /etc/fstab 2020-03-11T22:35:37.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:37.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_8 /etc/fstab 2020-03-11T22:35:37.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:37.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_9 /etc/fstab 2020-03-11T22:35:37.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:37.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_10 /etc/fstab 2020-03-11T22:35:37.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:37.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_11 /etc/fstab 2020-03-11T22:35:37.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:37.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_12 /etc/fstab 2020-03-11T22:35:37.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:37.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_13 /etc/fstab 2020-03-11T22:35:37.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:37.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_14 /etc/fstab 2020-03-11T22:35:37.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:37.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_15 /etc/fstab 2020-03-11T22:35:37.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:37.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_16 /etc/fstab 2020-03-11T22:35:38.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_17 /etc/fstab 2020-03-11T22:35:38.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_18 /etc/fstab 2020-03-11T22:35:38.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_19 /etc/fstab 2020-03-11T22:35:38.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_20 /etc/fstab 2020-03-11T22:35:38.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_21 /etc/fstab 2020-03-11T22:35:38.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_22 /etc/fstab 2020-03-11T22:35:38.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_23 /etc/fstab 2020-03-11T22:35:38.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_24 /etc/fstab 2020-03-11T22:35:38.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_25 /etc/fstab 2020-03-11T22:35:38.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_26 /etc/fstab 2020-03-11T22:35:38.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_27 /etc/fstab 2020-03-11T22:35:38.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_28 /etc/fstab 2020-03-11T22:35:38.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_29 /etc/fstab 2020-03-11T22:35:38.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_30 /etc/fstab 2020-03-11T22:35:38.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_31 /etc/fstab 2020-03-11T22:35:38.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:38.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_32 /etc/fstab 2020-03-11T22:35:39.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_33 /etc/fstab 2020-03-11T22:35:39.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_34 /etc/fstab 2020-03-11T22:35:39.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_35 /etc/fstab 2020-03-11T22:35:39.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_36 /etc/fstab 2020-03-11T22:35:39.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_37 /etc/fstab 2020-03-11T22:35:39.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_38 /etc/fstab 2020-03-11T22:35:39.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_39 /etc/fstab 2020-03-11T22:35:39.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_40 /etc/fstab 2020-03-11T22:35:39.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_41 /etc/fstab 2020-03-11T22:35:39.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_42 /etc/fstab 2020-03-11T22:35:39.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_43 /etc/fstab 2020-03-11T22:35:39.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_44 /etc/fstab 2020-03-11T22:35:39.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_45 /etc/fstab 2020-03-11T22:35:39.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_46 /etc/fstab 2020-03-11T22:35:39.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_47 /etc/fstab 2020-03-11T22:35:39.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_48 /etc/fstab 2020-03-11T22:35:39.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_49 /etc/fstab 2020-03-11T22:35:39.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_50 /etc/fstab 2020-03-11T22:35:39.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:39.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_51 /etc/fstab 2020-03-11T22:35:40.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_52 /etc/fstab 2020-03-11T22:35:40.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_53 /etc/fstab 2020-03-11T22:35:40.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_54 /etc/fstab 2020-03-11T22:35:40.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_55 /etc/fstab 2020-03-11T22:35:40.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_56 /etc/fstab 2020-03-11T22:35:40.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_57 /etc/fstab 2020-03-11T22:35:40.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_58 /etc/fstab 2020-03-11T22:35:40.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_59 /etc/fstab 2020-03-11T22:35:40.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_60 /etc/fstab 2020-03-11T22:35:40.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_61 /etc/fstab 2020-03-11T22:35:40.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_62 /etc/fstab 2020-03-11T22:35:40.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_63 /etc/fstab 2020-03-11T22:35:40.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_64 /etc/fstab 2020-03-11T22:35:40.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_65 /etc/fstab 2020-03-11T22:35:40.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:40.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_66 /etc/fstab 2020-03-11T22:35:41.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_67 /etc/fstab 2020-03-11T22:35:41.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_68 /etc/fstab 2020-03-11T22:35:41.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_69 /etc/fstab 2020-03-11T22:35:41.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_70 /etc/fstab 2020-03-11T22:35:41.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_71 /etc/fstab 2020-03-11T22:35:41.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_72 /etc/fstab 2020-03-11T22:35:41.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_73 /etc/fstab 2020-03-11T22:35:41.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_74 /etc/fstab 2020-03-11T22:35:41.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_75 /etc/fstab 2020-03-11T22:35:41.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_76 /etc/fstab 2020-03-11T22:35:41.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_77 /etc/fstab 2020-03-11T22:35:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_78 /etc/fstab 2020-03-11T22:35:41.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_79 /etc/fstab 2020-03-11T22:35:41.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_80 /etc/fstab 2020-03-11T22:35:41.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_81 /etc/fstab 2020-03-11T22:35:41.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:41.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_82 /etc/fstab 2020-03-11T22:35:42.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_83 /etc/fstab 2020-03-11T22:35:42.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_84 /etc/fstab 2020-03-11T22:35:42.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_85 /etc/fstab 2020-03-11T22:35:42.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_86 /etc/fstab 2020-03-11T22:35:42.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_87 /etc/fstab 2020-03-11T22:35:42.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_88 /etc/fstab 2020-03-11T22:35:42.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_89 /etc/fstab 2020-03-11T22:35:42.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_90 /etc/fstab 2020-03-11T22:35:42.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_91 /etc/fstab 2020-03-11T22:35:42.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_92 /etc/fstab 2020-03-11T22:35:42.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_93 /etc/fstab 2020-03-11T22:35:42.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_94 /etc/fstab 2020-03-11T22:35:42.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_95 /etc/fstab 2020-03-11T22:35:42.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:42.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_96 /etc/fstab 2020-03-11T22:35:43.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_97 /etc/fstab 2020-03-11T22:35:43.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_98 /etc/fstab 2020-03-11T22:35:43.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_99 /etc/fstab 2020-03-11T22:35:43.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:97: TEST_divergent: for i in $(seq 1 $testobjects) 2020-03-11T22:35:43.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:99: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:35:43.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:35:43.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:102: TEST_divergent: wait_for_clean 2020-03-11T22:35:43.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:35:43.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:35:43.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:35:43.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:35:43.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:35:43.179 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:35:43.179 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:35:43.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:35:43.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:35:43.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:35:43.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:35:43.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:35:43.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:35:43.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:35:43.235 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:35:43.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:35:43.583 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:35:43.583 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:35:43.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:35:43.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:43.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:35:43.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:35:43.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:35:43.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:43.714 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:35:43.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672973 2020-03-11T22:35:43.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672973' 2020-03-11T22:35:43.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:35:43.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:35:43.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=64424509450 2020-03-11T22:35:43.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-42949672973 2-64424509450' 2020-03-11T22:35:43.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:43.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:35:43.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:43.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:35:43.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:35:43.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:43.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:35:43.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:35:43.978 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:35:43.978 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:44.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836493 2020-03-11T22:35:44.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:35:45.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:35:45.383 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:35:45.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:35:45.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:45.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672973 2020-03-11T22:35:45.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:45.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:35:45.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672973 2020-03-11T22:35:45.748 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:45.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672973 2020-03-11T22:35:45.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2020-03-11T22:35:45.748 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:35:45.748 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672973 2020-03-11T22:35:46.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672973 -lt 42949672973 2020-03-11T22:35:46.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:35:46.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-64424509450 2020-03-11T22:35:46.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:35:46.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:35:46.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-64424509450 2020-03-11T22:35:46.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:35:46.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=64424509450 2020-03-11T22:35:46.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 64424509450' 2020-03-11T22:35:46.094 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 64424509450 2020-03-11T22:35:46.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:35:46.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 64424509451 -lt 64424509450 2020-03-11T22:35:46.438 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:35:46.439 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:46.439 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:46.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:35:46.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:35:46.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:35:46.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:35:46.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:35:46.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:35:46.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:35:46.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:35:47.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:35:47.209 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:35:47.210 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:35:47.210 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:35:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:35:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:35:47.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:35:47.651 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: get_pg test existing_1 2020-03-11T22:35:47.651 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:35:47.651 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:35:47.652 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:35:47.652 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:35:48.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:104: TEST_divergent: local pgid=1.0 2020-03-11T22:35:48.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:107: TEST_divergent: echo 'blackholing osds 0 2' 2020-03-11T22:35:48.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:108: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:48.003 INFO:tasks.workunit.client.0.smithi155.stdout:blackholing osds 0 2 2020-03-11T22:35:48.323 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:48.324 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:35:27.639709+0000 28'100 28:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:35:48.324 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:35:48.324 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:48.325 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:35:48.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:35:48.335 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.0 2020-03-11T22:35:48.335 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:35:48.335 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:35:48.335 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:48.335 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:48.335 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:48.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.0.asok 2020-03-11T22:35:48.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:35:48.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:35:48.424 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:35:48.424 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:35:48.424 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:35:48.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:109: TEST_divergent: for i in $non_divergent 2020-03-11T22:35:48.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: get_asok_path osd.2 2020-03-11T22:35:48.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:35:48.432 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:35:48.432 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:48.433 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:48.433 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:48.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.2.asok 2020-03-11T22:35:48.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: CEPH_ARGS= 2020-03-11T22:35:48.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:111: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:35:48.519 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:35:48.520 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:35:48.520 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:35:48.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:114: TEST_divergent: local case5=100 2020-03-11T22:35:48.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: expr 100 - 1 2020-03-11T22:35:48.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:115: TEST_divergent: local case3=99 2020-03-11T22:35:48.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:117: TEST_divergent: echo 'writing divergent object' 2020-03-11T22:35:48.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:119: TEST_divergent: echo 'create missing divergent object' 2020-03-11T22:35:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:118: TEST_divergent: rados -p test put existing_100 /etc/fstab 2020-03-11T22:35:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:120: TEST_divergent: inject_eio rep data test existing_99 td/divergent-priors 0 2020-03-11T22:35:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2226: inject_eio: local pooltype=rep 2020-03-11T22:35:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2227: inject_eio: shift 2020-03-11T22:35:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2228: inject_eio: local which=data 2020-03-11T22:35:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2229: inject_eio: shift 2020-03-11T22:35:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2230: inject_eio: local poolname=test 2020-03-11T22:35:48.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2231: inject_eio: shift 2020-03-11T22:35:48.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2232: inject_eio: local objname=existing_99 2020-03-11T22:35:48.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2233: inject_eio: shift 2020-03-11T22:35:48.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2234: inject_eio: local dir=td/divergent-priors 2020-03-11T22:35:48.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2235: inject_eio: shift 2020-03-11T22:35:48.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2236: inject_eio: local shard_id=0 2020-03-11T22:35:48.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2237: inject_eio: shift 2020-03-11T22:35:48.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: initial_osds=($(get_osds $poolname $objname)) 2020-03-11T22:35:48.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: get_osds test existing_99 2020-03-11T22:35:48.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:958: get_osds: local poolname=test 2020-03-11T22:35:48.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:959: get_osds: local objectname=existing_99 2020-03-11T22:35:48.533 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: ceph --format json osd map test existing_99 2020-03-11T22:35:48.533 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: jq '.acting | .[]' 2020-03-11T22:35:48.533 INFO:tasks.workunit.client.0.smithi155.stdout:writing divergent object 2020-03-11T22:35:48.533 INFO:tasks.workunit.client.0.smithi155.stdout:create missing divergent object 2020-03-11T22:35:48.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:962: get_osds: local 'osds=1 2020-03-11T22:35:48.881 INFO:tasks.workunit.client.0.smithi155.stderr:0 2020-03-11T22:35:48.881 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:35:48.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:964: get_osds: echo 1 0 2 2020-03-11T22:35:48.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2239: inject_eio: local -a initial_osds 2020-03-11T22:35:48.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2240: inject_eio: local osd_id=1 2020-03-11T22:35:48.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2241: inject_eio: '[' rep '!=' ec ']' 2020-03-11T22:35:48.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2242: inject_eio: shard_id= 2020-03-11T22:35:48.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: cat td/divergent-priors/1/type 2020-03-11T22:35:48.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2244: inject_eio: type=bluestore 2020-03-11T22:35:48.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2245: inject_eio: set_config osd 1 bluestore_debug_inject_read_err true 2020-03-11T22:35:48.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: set_config: local daemon=osd 2020-03-11T22:35:48.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: set_config: local id=1 2020-03-11T22:35:48.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: set_config: local config=bluestore_debug_inject_read_err 2020-03-11T22:35:48.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1101: set_config: local value=true 2020-03-11T22:35:48.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: jq 'has("success")' 2020-03-11T22:35:48.885 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: get_asok_path osd.1 2020-03-11T22:35:48.885 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:35:48.885 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:35:48.885 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:48.886 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:48.886 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:48.886 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.1.asok 2020-03-11T22:35:48.886 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: env CEPH_ARGS= ceph --format json daemon /tmp/ceph-asok.17464/ceph-osd.1.asok config set bluestore_debug_inject_read_err true 2020-03-11T22:35:48.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1105: set_config: test true == true 2020-03-11T22:35:48.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2246: inject_eio: local loop=0 2020-03-11T22:35:48.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2248: inject_eio: grep -q Invalid 2020-03-11T22:35:48.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: get_asok_path osd.1 2020-03-11T22:35:48.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:35:48.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:35:48.981 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:35:48.981 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:35:48.981 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:35:48.981 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.1.asok 2020-03-11T22:35:48.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: CEPH_ARGS= 2020-03-11T22:35:48.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2247: inject_eio: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.1.asok injectdataerr test existing_99 2020-03-11T22:35:49.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:122: TEST_divergent: sleep 10 2020-03-11T22:35:49.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:121: TEST_divergent: rados -p test get existing_99 td/divergent-priors/existing 2020-03-11T22:35:59.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:123: TEST_divergent: killall -9 rados 2020-03-11T22:35:59.095 INFO:tasks.workunit.client.0.smithi155.stderr:rados: no process found 2020-03-11T22:35:59.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:126: TEST_divergent: echo 'killing all the osds' 2020-03-11T22:35:59.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:127: TEST_divergent: ceph pg dump pgs 2020-03-11T22:35:59.095 INFO:tasks.workunit.client.0.smithi155.stdout:killing all the osds 2020-03-11T22:35:59.422 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:35:59.422 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 active+clean 2020-03-11T22:35:49.334886+0000 28'101 28:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:35:59.422 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:35:59.423 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:35:59.424 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:35:59.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:128: TEST_divergent: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:35:59.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:35:59.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:35:59.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:35:59.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:35:59.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:35:59.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:35:59.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:35:59.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.0 2020-03-11T22:36:00.053 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2020-03-11T22:36:00.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:36:00.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.1 2020-03-11T22:36:00.546 INFO:tasks.workunit.client.0.smithi155.stderr:osd.1 is already down. 2020-03-11T22:36:00.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:129: TEST_divergent: for i in $osds 2020-03-11T22:36:00.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:131: TEST_divergent: ceph osd down osd.2 2020-03-11T22:36:01.047 INFO:tasks.workunit.client.0.smithi155.stderr:osd.2 is already down. 2020-03-11T22:36:01.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:01.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.0 2020-03-11T22:36:01.549 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already out. 2020-03-11T22:36:01.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:133: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:01.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:135: TEST_divergent: ceph osd out osd.2 2020-03-11T22:36:02.050 INFO:tasks.workunit.client.0.smithi155.stderr:osd.2 is already out. 2020-03-11T22:36:02.058 INFO:tasks.workunit.client.0.smithi155.stdout:bringing up non_divergent 0 2 2020-03-11T22:36:02.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:139: TEST_divergent: echo 'bringing up non_divergent 0 2' 2020-03-11T22:36:02.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:140: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:02.379 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:02.379 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 0 0 0 48300 0 0 17 17 stale+active+clean 2020-03-11T22:35:49.334886+0000 28'101 28:121 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:36:02.379 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:36:02.379 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:02.381 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:36:02.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:02.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 0 2020-03-11T22:36:02.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:36:02.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:36:02.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:36:02.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:36:02.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:36:02.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:02.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:02.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:02.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:02.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:36:02.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:36:02.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:02.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:36:02.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:02.395 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:36:02.395 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:02.395 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:02.395 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:02.396 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:02.396 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:36:02.396 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:36:02.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:36:02.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:02.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:02.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:02.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:02.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:02.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:02.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:36:02.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:36:02.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:36:02.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:36:02.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:02.398 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:36:02.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:36:02.398 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:36:02.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:36:02.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:36:02.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:36:02.418 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:02.416+0000 7f1a4bfb9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:02.420 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:02.417+0000 7f1a4bfb9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:02.422 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:02.419+0000 7f1a4bfb9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:02.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:36:02.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:02.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:36:02.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:02.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:02.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:02.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:02.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:02.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:02.740 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:36:02.939 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:02.937+0000 7f1a4bfb9ec0 -1 Falling back to public interface 2020-03-11T22:36:03.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:03.841 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:03.839+0000 7f1a4bfb9ec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:36:04.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:04.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:04.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:04.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:04.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:04.090 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:36:04.446 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/22562,v1:127.0.0.1:6803/22562] [v2:127.0.0.1:6804/22562,v1:127.0.0.1:6805/22562] exists,up e73ac5a4-38f7-49d2-9ebe-aa4155e118b1 2020-03-11T22:36:04.446 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:36:04.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:04.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:04.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:04.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:141: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:04.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:143: TEST_divergent: activate_osd td/divergent-priors 2 2020-03-11T22:36:04.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:36:04.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:36:04.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:36:04.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:36:04.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:36:04.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:04.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:04.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:04.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:04.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:36:04.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:36:04.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:04.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:36:04.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:04.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:36:04.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:04.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:04.449 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:04.449 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:04.449 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:36:04.450 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:36:04.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:36:04.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:04.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:04.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:04.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:04.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:04.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:04.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:36:04.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:36:04.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:36:04.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:36:04.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:04.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:36:04.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:36:04.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:36:04.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:36:04.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:36:04.472 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:04.469+0000 7f990811cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:04.473 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:04.471+0000 7f990811cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:04.475 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:04.473+0000 7f990811cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:04.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:36:04.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:04.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:36:04.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:04.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:04.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:04.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:04.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:04.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:04.814 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:36:04.990 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:04.987+0000 7f990811cec0 -1 Falling back to public interface 2020-03-11T22:36:05.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:36:05.869 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:05.866+0000 7f990811cec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:36:06.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:36:06.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:06.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:36:06.161 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:36:06.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:06.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:06.555 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/22756,v1:127.0.0.1:6811/22756] [v2:127.0.0.1:6812/22756,v1:127.0.0.1:6813/22756] exists,up ea275d10-e3f1-4218-8e10-8f2a58b11e5b 2020-03-11T22:36:06.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:06.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:06.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:06.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:06.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.0 2020-03-11T22:36:07.083 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already in. 2020-03-11T22:36:07.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:145: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:07.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:147: TEST_divergent: ceph osd in osd.2 2020-03-11T22:36:07.725 INFO:tasks.workunit.client.0.smithi155.stderr:osd.2 is already in. 2020-03-11T22:36:07.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:36:07.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:150: TEST_divergent: wait_for_clean 2020-03-11T22:36:07.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:36:07.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:36:07.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:36:07.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:36:07.733 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:36:07.734 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:36:07.734 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:36:07.734 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:36:07.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:36:07.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:36:07.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:36:07.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:36:07.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:36:07.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:36:07.789 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:36:08.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:36:08.131 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:36:08.131 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:36:08.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:36:08.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:08.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:36:08.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:36:08.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067' 2020-03-11T22:36:08.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:08.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:36:08.354 INFO:tasks.workunit.client.0.smithi155.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:36:08.362 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:36:08.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1-' 2020-03-11T22:36:08.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:36:08.363 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:36:08.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:36:08.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1- 2-150323855363' 2020-03-11T22:36:08.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:08.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888067 2020-03-11T22:36:08.495 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:08.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:36:08.496 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888067 2020-03-11T22:36:08.496 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:08.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:36:08.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2020-03-11T22:36:08.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:08.497 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 146028888067 2020-03-11T22:36:08.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-11T22:36:08.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:09.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:36:09.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:10.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-11T22:36:10.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:36:11.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:36:11.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:36:11.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888068 -lt 146028888067 2020-03-11T22:36:11.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:11.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:36:11.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:11.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:36:11.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:36:11.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:11.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:36:11.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:36:11.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:36:11.543 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 2020-03-11T22:36:11.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672976 -lt 2020-03-11T22:36:11.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 42949672976: unary operator expected 2020-03-11T22:36:11.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:36:11.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855363 2020-03-11T22:36:11.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:36:11.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:36:11.889 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855363 2020-03-11T22:36:11.889 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:36:11.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:36:11.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-03-11T22:36:11.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:36:11.891 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 150323855363 2020-03-11T22:36:12.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:36:12.243 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:36:12.243 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:12.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:12.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:36:12.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:12.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:12.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:12.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:12.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:12.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:12.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:13.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:13.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:13.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:13.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:13.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:13.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:36:13.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:36:13.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:36:13.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:13.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:36:13.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:13.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:13.568 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:13.568 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:13.569 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:13.569 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:13.569 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:13.569 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:13.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:13.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:13.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:13.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:14.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:14.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:14.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:14.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:14.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:14.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:14.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:14.347 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:14.347 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:14.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:14.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:14.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:36:14.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:14.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:36:14.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:14.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:14.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:14.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:14.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:14.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:14.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:14.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:15.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:15.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:15.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:15.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:15.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:15.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:15.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:15.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:15.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:15.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:15.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:15.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:15.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:16.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:16.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:16.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:36:16.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:16.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:36:16.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:16.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:16.640 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:16.640 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:16.640 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:16.640 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:16.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:16.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:16.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:16.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:16.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:16.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:17.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:17.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:17.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:17.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:17.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:17.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:17.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:17.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:17.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:17.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:17.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:17.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:36:17.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:17.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:36:18.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:18.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:18.656 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:18.657 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:18.657 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:18.657 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:18.657 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:18.657 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:18.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:18.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:18.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:18.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:19.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:19.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:19.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:19.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:19.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:19.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:19.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:19.442 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:19.443 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:19.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:19.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:19.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:36:19.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:19.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:36:21.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:21.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:21.510 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:21.510 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:21.511 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:21.511 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:21.511 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:21.511 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:21.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:21.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:21.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:21.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:22.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:22.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:22.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:22.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:22.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:22.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:22.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:22.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:22.300 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:22.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:22.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:22.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:36:22.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:22.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:36:25.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:25.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:25.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:25.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:25.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:25.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:25.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:25.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:26.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:26.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:26.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:26.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:26.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:26.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:26.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:26.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:26.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:26.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:26.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:26.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:27.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:27.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:27.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:36:27.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:27.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:36:33.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:33.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:33.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:33.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:33.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:33.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:33.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:33.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:33.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:33.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:33.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:33.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:34.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:34.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:34.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:34.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:34.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:34.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:34.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:34.374 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:34.374 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:34.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:34.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:34.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:36:34.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:36:34.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:36:42.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:36:42.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:36:42.115 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:36:42.115 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:36:42.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:36:42.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:36:42.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:36:42.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:36:42.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:36:42.464 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:36:42.465 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:36:42.465 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:36:42.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:36:42.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:36:42.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:36:42.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:36:42.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:36:42.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:36:42.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:36:42.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:36:42.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:36:43.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:36:43.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:36:43.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:36:43.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:36:43.795 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:36:43.796 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_fingerprint": "42f1d474-02e5-4144-80ae-1402767c171e", 2020-03-11T22:36:43.796 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:36:43.796 INFO:tasks.workunit.client.0.smithi155.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:36:43.796 INFO:tasks.workunit.client.0.smithi155.stdout: "timestamp": "2020-03-11T22:36:43.792594+0000", 2020-03-11T22:36:43.796 INFO:tasks.workunit.client.0.smithi155.stdout: "tag": "", 2020-03-11T22:36:43.796 INFO:tasks.workunit.client.0.smithi155.stdout: "health": { 2020-03-11T22:36:43.797 INFO:tasks.workunit.client.0.smithi155.stdout: "status": "HEALTH_WARN", 2020-03-11T22:36:43.797 INFO:tasks.workunit.client.0.smithi155.stdout: "checks": { 2020-03-11T22:36:43.797 INFO:tasks.workunit.client.0.smithi155.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:36:43.797 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:36:43.797 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:36:43.797 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:36:43.797 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 17 2020-03-11T22:36:43.798 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.798 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [], 2020-03-11T22:36:43.798 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:36:43.798 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.798 INFO:tasks.workunit.client.0.smithi155.stdout: "OSD_DOWN": { 2020-03-11T22:36:43.798 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:36:43.798 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:36:43.798 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 osds down", 2020-03-11T22:36:43.799 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:36:43.799 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.799 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:36:43.799 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.799 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "osd.1 (root=default,host=smithi155) is down" 2020-03-11T22:36:43.799 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.799 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.800 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:36:43.800 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.800 INFO:tasks.workunit.client.0.smithi155.stdout: "PG_DEGRADED": { 2020-03-11T22:36:43.800 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:36:43.800 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:36:43.800 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:36:43.800 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:36:43.800 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.801 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:36:43.801 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.801 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:36:43.801 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.801 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.801 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:36:43.801 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.802 INFO:tasks.workunit.client.0.smithi155.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:36:43.802 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:36:43.802 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:36:43.802 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:36:43.802 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:36:43.802 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.802 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:36:43.803 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.803 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:36:43.803 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.803 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.803 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:36:43.803 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.803 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.803 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:36:43.804 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.804 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.804 INFO:tasks.workunit.client.0.smithi155.stdout: "mutes": [] 2020-03-11T22:36:43.804 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.804 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_first_committed": 1, 2020-03-11T22:36:43.804 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_last_committed": 1, 2020-03-11T22:36:43.804 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap": { 2020-03-11T22:36:43.804 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:36:43.805 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:36:43.805 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:35:05.065808Z", 2020-03-11T22:36:43.805 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:35:05.065808Z", 2020-03-11T22:36:43.805 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release": 15, 2020-03-11T22:36:43.805 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:36:43.805 INFO:tasks.workunit.client.0.smithi155.stdout: "features": { 2020-03-11T22:36:43.805 INFO:tasks.workunit.client.0.smithi155.stdout: "persistent": [ 2020-03-11T22:36:43.806 INFO:tasks.workunit.client.0.smithi155.stdout: "kraken", 2020-03-11T22:36:43.806 INFO:tasks.workunit.client.0.smithi155.stdout: "luminous", 2020-03-11T22:36:43.806 INFO:tasks.workunit.client.0.smithi155.stdout: "mimic", 2020-03-11T22:36:43.806 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap-prune", 2020-03-11T22:36:43.806 INFO:tasks.workunit.client.0.smithi155.stdout: "nautilus", 2020-03-11T22:36:43.806 INFO:tasks.workunit.client.0.smithi155.stdout: "octopus" 2020-03-11T22:36:43.806 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.806 INFO:tasks.workunit.client.0.smithi155.stdout: "optional": [] 2020-03-11T22:36:43.807 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.807 INFO:tasks.workunit.client.0.smithi155.stdout: "mons": [ 2020-03-11T22:36:43.807 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.807 INFO:tasks.workunit.client.0.smithi155.stdout: "rank": 0, 2020-03-11T22:36:43.807 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "a", 2020-03-11T22:36:43.807 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:36:43.807 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.808 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.808 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.808 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:36:43.808 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 0 2020-03-11T22:36:43.808 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.808 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.808 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.808 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:36:43.809 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:36:43.809 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 0, 2020-03-11T22:36:43.809 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 0 2020-03-11T22:36:43.809 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.809 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.809 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.809 INFO:tasks.workunit.client.0.smithi155.stdout: "quorum": [ 2020-03-11T22:36:43.809 INFO:tasks.workunit.client.0.smithi155.stdout: 0 2020-03-11T22:36:43.810 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.810 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap": { 2020-03-11T22:36:43.810 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 38, 2020-03-11T22:36:43.810 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:36:43.810 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:35:05.814329+0000", 2020-03-11T22:36:43.810 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:36:07.610650+0000", 2020-03-11T22:36:43.810 INFO:tasks.workunit.client.0.smithi155.stdout: "last_up_change": "2020-03-11T22:36:05.949083+0000", 2020-03-11T22:36:43.811 INFO:tasks.workunit.client.0.smithi155.stdout: "last_in_change": "2020-03-11T22:36:07.376878+0000", 2020-03-11T22:36:43.811 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:36:43.811 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_num": 5800384, 2020-03-11T22:36:43.811 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_set": [ 2020-03-11T22:36:43.811 INFO:tasks.workunit.client.0.smithi155.stdout: "nodown", 2020-03-11T22:36:43.811 INFO:tasks.workunit.client.0.smithi155.stdout: "noin", 2020-03-11T22:36:43.811 INFO:tasks.workunit.client.0.smithi155.stdout: "noout", 2020-03-11T22:36:43.811 INFO:tasks.workunit.client.0.smithi155.stdout: "pglog_hardlimit", 2020-03-11T22:36:43.812 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snapdirs", 2020-03-11T22:36:43.812 INFO:tasks.workunit.client.0.smithi155.stdout: "recovery_deletes", 2020-03-11T22:36:43.812 INFO:tasks.workunit.client.0.smithi155.stdout: "sortbitwise" 2020-03-11T22:36:43.812 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.812 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_version": 7, 2020-03-11T22:36:43.812 INFO:tasks.workunit.client.0.smithi155.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:36:43.812 INFO:tasks.workunit.client.0.smithi155.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:36:43.812 INFO:tasks.workunit.client.0.smithi155.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:36:43.813 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_snapshot": "", 2020-03-11T22:36:43.813 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_max": 1, 2020-03-11T22:36:43.813 INFO:tasks.workunit.client.0.smithi155.stdout: "max_osd": 3, 2020-03-11T22:36:43.813 INFO:tasks.workunit.client.0.smithi155.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:36:43.813 INFO:tasks.workunit.client.0.smithi155.stdout: "min_compat_client": "jewel", 2020-03-11T22:36:43.813 INFO:tasks.workunit.client.0.smithi155.stdout: "require_osd_release": "octopus", 2020-03-11T22:36:43.813 INFO:tasks.workunit.client.0.smithi155.stdout: "pools": [ 2020-03-11T22:36:43.814 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.814 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:36:43.815 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_name": "test", 2020-03-11T22:36:43.815 INFO:tasks.workunit.client.0.smithi155.stdout: "create_time": "2020-03-11T22:35:26.753573+0000", 2020-03-11T22:36:43.815 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": 1, 2020-03-11T22:36:43.815 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_names": "hashpspool", 2020-03-11T22:36:43.815 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2020-03-11T22:36:43.815 INFO:tasks.workunit.client.0.smithi155.stdout: "size": 3, 2020-03-11T22:36:43.816 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 2, 2020-03-11T22:36:43.816 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_rule": 0, 2020-03-11T22:36:43.816 INFO:tasks.workunit.client.0.smithi155.stdout: "object_hash": 2, 2020-03-11T22:36:43.816 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:36:43.816 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num": 1, 2020-03-11T22:36:43.816 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num": 1, 2020-03-11T22:36:43.816 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num_target": 1, 2020-03-11T22:36:43.816 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_target": 1, 2020-03-11T22:36:43.817 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_pending": 1, 2020-03-11T22:36:43.817 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pg_merge_meta": { 2020-03-11T22:36:43.817 INFO:tasks.workunit.client.0.smithi155.stdout: "source_pgid": "0.0", 2020-03-11T22:36:43.817 INFO:tasks.workunit.client.0.smithi155.stdout: "ready_epoch": 0, 2020-03-11T22:36:43.817 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_started": 0, 2020-03-11T22:36:43.817 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_clean": 0, 2020-03-11T22:36:43.817 INFO:tasks.workunit.client.0.smithi155.stdout: "source_version": "0'0", 2020-03-11T22:36:43.817 INFO:tasks.workunit.client.0.smithi155.stdout: "target_version": "0'0" 2020-03-11T22:36:43.818 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.818 INFO:tasks.workunit.client.0.smithi155.stdout: "last_change": "28", 2020-03-11T22:36:43.818 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend": "0", 2020-03-11T22:36:43.818 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:36:43.818 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:36:43.818 INFO:tasks.workunit.client.0.smithi155.stdout: "auid": 0, 2020-03-11T22:36:43.818 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:36:43.819 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_seq": 0, 2020-03-11T22:36:43.819 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_epoch": 0, 2020-03-11T22:36:43.819 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_snaps": [], 2020-03-11T22:36:43.819 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps": "[]", 2020-03-11T22:36:43.819 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_bytes": 0, 2020-03-11T22:36:43.819 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_objects": 0, 2020-03-11T22:36:43.819 INFO:tasks.workunit.client.0.smithi155.stdout: "tiers": [], 2020-03-11T22:36:43.819 INFO:tasks.workunit.client.0.smithi155.stdout: "tier_of": -1, 2020-03-11T22:36:43.820 INFO:tasks.workunit.client.0.smithi155.stdout: "read_tier": -1, 2020-03-11T22:36:43.820 INFO:tasks.workunit.client.0.smithi155.stdout: "write_tier": -1, 2020-03-11T22:36:43.820 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_mode": "none", 2020-03-11T22:36:43.820 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_bytes": 0, 2020-03-11T22:36:43.820 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_objects": 0, 2020-03-11T22:36:43.820 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:36:43.820 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:36:43.820 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:36:43.821 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_flush_age": 0, 2020-03-11T22:36:43.821 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_evict_age": 0, 2020-03-11T22:36:43.821 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profile": "", 2020-03-11T22:36:43.821 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_params": { 2020-03-11T22:36:43.821 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "none" 2020-03-11T22:36:43.821 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.821 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_period": 0, 2020-03-11T22:36:43.822 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_count": 0, 2020-03-11T22:36:43.822 INFO:tasks.workunit.client.0.smithi155.stdout: "use_gmt_hitset": true, 2020-03-11T22:36:43.822 INFO:tasks.workunit.client.0.smithi155.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:36:43.822 INFO:tasks.workunit.client.0.smithi155.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:36:43.822 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:36:43.822 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:36:43.822 INFO:tasks.workunit.client.0.smithi155.stdout: "grade_table": [], 2020-03-11T22:36:43.822 INFO:tasks.workunit.client.0.smithi155.stdout: "stripe_width": 0, 2020-03-11T22:36:43.823 INFO:tasks.workunit.client.0.smithi155.stdout: "expected_num_objects": 0, 2020-03-11T22:36:43.823 INFO:tasks.workunit.client.0.smithi155.stdout: "fast_read": false, 2020-03-11T22:36:43.823 INFO:tasks.workunit.client.0.smithi155.stdout: "options": {}, 2020-03-11T22:36:43.823 INFO:tasks.workunit.client.0.smithi155.stdout: "application_metadata": {} 2020-03-11T22:36:43.823 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.823 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.823 INFO:tasks.workunit.client.0.smithi155.stdout: "osds": [ 2020-03-11T22:36:43.823 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.824 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:36:43.824 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "e73ac5a4-38f7-49d2-9ebe-aa4155e118b1", 2020-03-11T22:36:43.824 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:36:43.824 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:36:43.824 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:36:43.824 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:36:43.824 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 5, 2020-03-11T22:36:43.824 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 28, 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 34, 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 37, 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 29, 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.825 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.826 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.826 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:36:43.826 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22562 2020-03-11T22:36:43.826 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.826 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.826 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:36:43.826 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:36:43.827 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22562 2020-03-11T22:36:43.827 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.827 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.827 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.827 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:36:43.827 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.827 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.828 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.828 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:36:43.828 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22562 2020-03-11T22:36:43.828 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.828 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.828 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:36:43.828 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:36:43.829 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22562 2020-03-11T22:36:43.829 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.829 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.829 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.829 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:36:43.829 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.829 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.829 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.830 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:36:43.830 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22562 2020-03-11T22:36:43.830 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.830 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.830 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:36:43.830 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:36:43.830 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22562 2020-03-11T22:36:43.831 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.831 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.831 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.831 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:36:43.831 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.831 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.831 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.831 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:36:43.832 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22562 2020-03-11T22:36:43.832 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.832 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.832 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:36:43.832 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:36:43.832 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22562 2020-03-11T22:36:43.832 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.833 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.833 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.833 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6803/22562", 2020-03-11T22:36:43.833 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6805/22562", 2020-03-11T22:36:43.833 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6809/22562", 2020-03-11T22:36:43.833 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6807/22562", 2020-03-11T22:36:43.833 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:36:43.833 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:36:43.834 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:36:43.834 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.834 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.834 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.834 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:36:43.834 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "748827f2-8dc8-4162-b402-942b80b4efe3", 2020-03-11T22:36:43.834 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 0, 2020-03-11T22:36:43.834 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:36:43.835 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:36:43.835 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:36:43.835 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 0, 2020-03-11T22:36:43.835 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 0, 2020-03-11T22:36:43.835 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 10, 2020-03-11T22:36:43.835 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 22, 2020-03-11T22:36:43.835 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 30, 2020-03-11T22:36:43.835 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:36:43.836 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:36:43.836 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.836 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.836 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.836 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:36:43.836 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 18256 2020-03-11T22:36:43.836 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.837 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.837 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:36:43.837 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:36:43.837 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 18256 2020-03-11T22:36:43.837 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.837 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.837 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.837 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:36:43.838 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.838 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.838 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.838 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:36:43.838 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 18256 2020-03-11T22:36:43.838 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.838 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.839 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:36:43.839 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:36:43.839 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 18256 2020-03-11T22:36:43.839 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.839 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.839 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.839 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:36:43.839 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.840 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.840 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.840 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:36:43.840 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 18256 2020-03-11T22:36:43.840 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.840 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.840 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:36:43.840 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:36:43.841 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 18256 2020-03-11T22:36:43.841 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.841 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.841 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.841 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:36:43.841 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.841 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.842 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.842 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:36:43.842 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 18256 2020-03-11T22:36:43.842 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.842 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.842 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:36:43.842 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:36:43.842 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 18256 2020-03-11T22:36:43.843 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.843 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.843 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.843 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/18256", 2020-03-11T22:36:43.843 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/18256", 2020-03-11T22:36:43.843 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/18256", 2020-03-11T22:36:43.843 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/18256", 2020-03-11T22:36:43.843 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:36:43.844 INFO:tasks.workunit.client.0.smithi155.stdout: "exists" 2020-03-11T22:36:43.844 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.844 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.844 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.844 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:36:43.844 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "ea275d10-e3f1-4218-8e10-8f2a58b11e5b", 2020-03-11T22:36:43.844 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:36:43.844 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:36:43.845 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:36:43.845 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:36:43.845 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 15, 2020-03-11T22:36:43.845 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 28, 2020-03-11T22:36:43.845 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 35, 2020-03-11T22:36:43.845 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 0, 2020-03-11T22:36:43.845 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 31, 2020-03-11T22:36:43.846 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:36:43.846 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:36:43.846 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.846 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.846 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.846 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:36:43.846 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22756 2020-03-11T22:36:43.847 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.847 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.847 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:36:43.847 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:36:43.847 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22756 2020-03-11T22:36:43.847 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.847 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.847 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.848 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:36:43.848 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.848 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.848 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.848 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:36:43.848 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22756 2020-03-11T22:36:43.848 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.848 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.849 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:36:43.849 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:36:43.849 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22756 2020-03-11T22:36:43.849 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.849 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.849 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.849 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:36:43.850 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.850 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.850 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.850 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:36:43.850 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22756 2020-03-11T22:36:43.850 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.850 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.850 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:36:43.851 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:36:43.851 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22756 2020-03-11T22:36:43.851 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.851 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.851 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.851 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:36:43.851 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:36:43.852 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.852 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:36:43.852 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:36:43.852 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22756 2020-03-11T22:36:43.852 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.852 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.852 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:36:43.852 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:36:43.853 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 22756 2020-03-11T22:36:43.853 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.853 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.853 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.853 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/22756", 2020-03-11T22:36:43.853 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/22756", 2020-03-11T22:36:43.853 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/22756", 2020-03-11T22:36:43.853 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/22756", 2020-03-11T22:36:43.854 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:36:43.854 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:36:43.854 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:36:43.854 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.854 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.854 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.854 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_xinfo": [ 2020-03-11T22:36:43.855 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.855 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:36:43.855 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:35:59.835029+0000", 2020-03-11T22:36:43.855 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:36:43.855 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:36:43.855 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:36:43.855 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:36:43.855 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:35:12.256447+0000", 2020-03-11T22:36:43.856 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:36:43.856 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.856 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.856 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:36:43.856 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:36:00.359051+0000", 2020-03-11T22:36:43.857 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:36:43.857 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:36:43.857 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:36:43.857 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:36:43.857 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:35:18.179290+0000", 2020-03-11T22:36:43.857 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:36:43.857 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.858 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.858 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:36:43.858 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:36:00.850440+0000", 2020-03-11T22:36:43.858 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:36:43.858 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:36:43.858 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:36:43.858 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:36:43.858 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:35:24.405118+0000", 2020-03-11T22:36:43.859 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:36:43.859 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.859 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.859 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap": [], 2020-03-11T22:36:43.859 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap_items": [], 2020-03-11T22:36:43.859 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_temp": [], 2020-03-11T22:36:43.859 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_temp": [], 2020-03-11T22:36:43.859 INFO:tasks.workunit.client.0.smithi155.stdout: "blacklist": {}, 2020-03-11T22:36:43.860 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profiles": { 2020-03-11T22:36:43.860 INFO:tasks.workunit.client.0.smithi155.stdout: "default": { 2020-03-11T22:36:43.860 INFO:tasks.workunit.client.0.smithi155.stdout: "k": "2", 2020-03-11T22:36:43.860 INFO:tasks.workunit.client.0.smithi155.stdout: "m": "2", 2020-03-11T22:36:43.860 INFO:tasks.workunit.client.0.smithi155.stdout: "plugin": "jerasure", 2020-03-11T22:36:43.860 INFO:tasks.workunit.client.0.smithi155.stdout: "technique": "reed_sol_van" 2020-03-11T22:36:43.860 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.861 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.861 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps_queue": [], 2020-03-11T22:36:43.861 INFO:tasks.workunit.client.0.smithi155.stdout: "new_removed_snaps": [], 2020-03-11T22:36:43.861 INFO:tasks.workunit.client.0.smithi155.stdout: "new_purged_snaps": [], 2020-03-11T22:36:43.861 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_node_flags": {}, 2020-03-11T22:36:43.861 INFO:tasks.workunit.client.0.smithi155.stdout: "device_class_flags": {} 2020-03-11T22:36:43.861 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.861 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_metadata": [ 2020-03-11T22:36:43.862 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.862 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:36:43.862 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:36:43.862 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6804/22562,v1:127.0.0.1:6805/22562]", 2020-03-11T22:36:43.862 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:36:43.862 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:36:43.862 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:36:43.863 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:36:43.863 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:36:43.863 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:36:43.863 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:36:43.863 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:36:43.863 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:36:43.863 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:36:43.864 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:36:43.864 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:36:43.864 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:36:43.864 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:36:43.864 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:36:43.864 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:36:43.864 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:36:43.864 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:36:43.865 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:36:43.865 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:36:43.865 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:36:43.865 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:36:43.865 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:36:43.865 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:36:43.865 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:36:43.865 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6802/22562,v1:127.0.0.1:6803/22562]", 2020-03-11T22:36:43.866 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:36:43.866 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/22562,v1:127.0.0.1:6809/22562]", 2020-03-11T22:36:43.866 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/22562,v1:127.0.0.1:6807/22562]", 2020-03-11T22:36:43.866 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:36:43.866 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:36:43.866 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:36:43.866 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:36:43.867 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:36:43.867 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:36:43.867 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:36:43.867 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:36:43.867 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:36:43.867 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:36:43.867 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:36:43.867 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:36:43.868 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.868 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.868 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:36:43.868 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:36:43.868 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/18256,v1:127.0.0.1:6813/18256]", 2020-03-11T22:36:43.868 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:36:43.868 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:36:43.869 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:36:43.869 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:36:43.869 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:36:43.869 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:36:43.869 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:36:43.869 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:36:43.869 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:36:43.869 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:36:43.870 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:36:43.870 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:36:43.870 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:36:43.870 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:36:43.870 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:36:43.870 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:36:43.870 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:36:43.871 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:36:43.871 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:36:43.871 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:36:43.871 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:36:43.871 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:36:43.871 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:36:43.871 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:36:43.871 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:36:43.872 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/18256,v1:127.0.0.1:6811/18256]", 2020-03-11T22:36:43.872 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:36:43.872 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/18256,v1:127.0.0.1:6817/18256]", 2020-03-11T22:36:43.872 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/18256,v1:127.0.0.1:6815/18256]", 2020-03-11T22:36:43.872 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:36:43.872 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:36:43.872 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:36:43.873 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:36:43.873 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:36:43.873 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:36:43.873 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:36:43.873 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:36:43.873 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:36:43.873 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:36:43.874 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:36:43.874 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:36:43.874 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.874 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.874 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:36:43.874 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:36:43.874 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/22756,v1:127.0.0.1:6813/22756]", 2020-03-11T22:36:43.875 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:36:43.875 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:36:43.875 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:36:43.875 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:36:43.875 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:36:43.875 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:36:43.875 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:36:43.875 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:36:43.876 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:36:43.876 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:36:43.876 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:36:43.876 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:36:43.876 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:36:43.876 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:36:43.876 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:36:43.876 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:36:43.877 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:36:43.877 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:36:43.877 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:36:43.877 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:36:43.877 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:36:43.877 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:36:43.877 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:36:43.878 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:36:43.878 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:36:43.878 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/22756,v1:127.0.0.1:6811/22756]", 2020-03-11T22:36:43.878 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:36:43.878 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/22756,v1:127.0.0.1:6817/22756]", 2020-03-11T22:36:43.878 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/22756,v1:127.0.0.1:6815/22756]", 2020-03-11T22:36:43.878 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:36:43.878 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:36:43.879 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:36:43.879 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:36:43.879 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:36:43.879 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:36:43.879 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:36:43.879 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:36:43.879 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:36:43.879 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:36:43.880 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:36:43.880 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:36:43.880 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.880 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.880 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_first_committed": 1, 2020-03-11T22:36:43.880 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_last_committed": 38, 2020-03-11T22:36:43.880 INFO:tasks.workunit.client.0.smithi155.stdout: "crushmap": { 2020-03-11T22:36:43.881 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": [ 2020-03-11T22:36:43.881 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.881 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:36:43.881 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.0", 2020-03-11T22:36:43.881 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:36:43.881 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.881 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.881 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:36:43.882 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.1", 2020-03-11T22:36:43.882 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:36:43.882 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.882 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.882 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:36:43.882 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.2", 2020-03-11T22:36:43.882 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:36:43.883 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.883 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.883 INFO:tasks.workunit.client.0.smithi155.stdout: "types": [ 2020-03-11T22:36:43.883 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.883 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 0, 2020-03-11T22:36:43.883 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd" 2020-03-11T22:36:43.883 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.884 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.884 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:36:43.884 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "host" 2020-03-11T22:36:43.884 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.884 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.884 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 2, 2020-03-11T22:36:43.884 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "chassis" 2020-03-11T22:36:43.885 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.885 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.885 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 3, 2020-03-11T22:36:43.885 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "rack" 2020-03-11T22:36:43.885 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.885 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.886 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 4, 2020-03-11T22:36:43.886 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "row" 2020-03-11T22:36:43.886 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.886 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.886 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 5, 2020-03-11T22:36:43.886 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pdu" 2020-03-11T22:36:43.886 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.886 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.887 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 6, 2020-03-11T22:36:43.887 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pod" 2020-03-11T22:36:43.887 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.887 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.887 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 7, 2020-03-11T22:36:43.887 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "room" 2020-03-11T22:36:43.887 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.887 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.888 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 8, 2020-03-11T22:36:43.888 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "datacenter" 2020-03-11T22:36:43.888 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.888 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.888 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 9, 2020-03-11T22:36:43.888 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "zone" 2020-03-11T22:36:43.888 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.889 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.889 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 10, 2020-03-11T22:36:43.889 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "region" 2020-03-11T22:36:43.889 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.889 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.889 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:36:43.889 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "root" 2020-03-11T22:36:43.889 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.890 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.890 INFO:tasks.workunit.client.0.smithi155.stdout: "buckets": [ 2020-03-11T22:36:43.890 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.890 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -1, 2020-03-11T22:36:43.890 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default", 2020-03-11T22:36:43.890 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:36:43.890 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:36:43.891 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:36:43.891 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:36:43.891 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:36:43.891 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:36:43.891 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.891 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:36:43.891 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:36:43.891 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:36:43.892 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.892 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.892 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.892 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.892 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -2, 2020-03-11T22:36:43.892 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default~hdd", 2020-03-11T22:36:43.892 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:36:43.892 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:36:43.893 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:36:43.893 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:36:43.893 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:36:43.893 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:36:43.893 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.893 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:36:43.893 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:36:43.893 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:36:43.894 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.894 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.894 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.894 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.894 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:36:43.894 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155", 2020-03-11T22:36:43.894 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:36:43.894 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:36:43.895 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:36:43.895 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:36:43.895 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:36:43.895 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:36:43.895 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.895 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:36:43.895 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:36:43.896 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:36:43.896 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.896 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.896 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:36:43.896 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:36:43.896 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:36:43.896 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.896 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.897 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:36:43.897 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:36:43.897 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:36:43.897 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.897 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.897 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.897 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.898 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:36:43.898 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155~hdd", 2020-03-11T22:36:43.898 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:36:43.898 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:36:43.898 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:36:43.898 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:36:43.898 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:36:43.898 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:36:43.899 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.899 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:36:43.899 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:36:43.899 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:36:43.899 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.899 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.899 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:36:43.899 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:36:43.900 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:36:43.900 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.900 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.900 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:36:43.900 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:36:43.900 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:36:43.900 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.901 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.901 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.901 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.901 INFO:tasks.workunit.client.0.smithi155.stdout: "rules": [ 2020-03-11T22:36:43.901 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.901 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_id": 0, 2020-03-11T22:36:43.901 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_name": "replicated_rule", 2020-03-11T22:36:43.901 INFO:tasks.workunit.client.0.smithi155.stdout: "ruleset": 0, 2020-03-11T22:36:43.902 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2020-03-11T22:36:43.902 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 1, 2020-03-11T22:36:43.902 INFO:tasks.workunit.client.0.smithi155.stdout: "max_size": 10, 2020-03-11T22:36:43.902 INFO:tasks.workunit.client.0.smithi155.stdout: "steps": [ 2020-03-11T22:36:43.902 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.902 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "take", 2020-03-11T22:36:43.902 INFO:tasks.workunit.client.0.smithi155.stdout: "item": -1, 2020-03-11T22:36:43.903 INFO:tasks.workunit.client.0.smithi155.stdout: "item_name": "default" 2020-03-11T22:36:43.903 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.903 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.903 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "choose_firstn", 2020-03-11T22:36:43.903 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 0, 2020-03-11T22:36:43.903 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "osd" 2020-03-11T22:36:43.903 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.903 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.904 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "emit" 2020-03-11T22:36:43.904 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.904 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:36:43.904 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.904 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.904 INFO:tasks.workunit.client.0.smithi155.stdout: "tunables": { 2020-03-11T22:36:43.904 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_tries": 0, 2020-03-11T22:36:43.904 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:36:43.905 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_total_tries": 50, 2020-03-11T22:36:43.905 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:36:43.905 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:36:43.905 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_stable": 1, 2020-03-11T22:36:43.905 INFO:tasks.workunit.client.0.smithi155.stdout: "straw_calc_version": 1, 2020-03-11T22:36:43.905 INFO:tasks.workunit.client.0.smithi155.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:36:43.905 INFO:tasks.workunit.client.0.smithi155.stdout: "profile": "jewel", 2020-03-11T22:36:43.905 INFO:tasks.workunit.client.0.smithi155.stdout: "optimal_tunables": 1, 2020-03-11T22:36:43.906 INFO:tasks.workunit.client.0.smithi155.stdout: "legacy_tunables": 0, 2020-03-11T22:36:43.906 INFO:tasks.workunit.client.0.smithi155.stdout: "minimum_required_version": "jewel", 2020-03-11T22:36:43.906 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables": 1, 2020-03-11T22:36:43.906 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables2": 1, 2020-03-11T22:36:43.906 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v2_rules": 0, 2020-03-11T22:36:43.906 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables3": 1, 2020-03-11T22:36:43.906 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v3_rules": 0, 2020-03-11T22:36:43.907 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v4_buckets": 1, 2020-03-11T22:36:43.907 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables5": 1, 2020-03-11T22:36:43.907 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v5_rules": 0 2020-03-11T22:36:43.907 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.907 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_args": {} 2020-03-11T22:36:43.907 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.907 INFO:tasks.workunit.client.0.smithi155.stdout: "fsmap": { 2020-03-11T22:36:43.907 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:36:43.908 INFO:tasks.workunit.client.0.smithi155.stdout: "default_fscid": -1, 2020-03-11T22:36:43.908 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": { 2020-03-11T22:36:43.908 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": {}, 2020-03-11T22:36:43.908 INFO:tasks.workunit.client.0.smithi155.stdout: "ro_compat": {}, 2020-03-11T22:36:43.908 INFO:tasks.workunit.client.0.smithi155.stdout: "incompat": { 2020-03-11T22:36:43.908 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_1": "base v0.20", 2020-03-11T22:36:43.908 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:36:43.908 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:36:43.909 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:36:43.909 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:36:43.909 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:36:43.909 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_8": "no anchor table", 2020-03-11T22:36:43.909 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_9": "file layout v2", 2020-03-11T22:36:43.909 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:36:43.909 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.910 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.910 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_flags": { 2020-03-11T22:36:43.910 INFO:tasks.workunit.client.0.smithi155.stdout: "enable_multiple": false, 2020-03-11T22:36:43.910 INFO:tasks.workunit.client.0.smithi155.stdout: "ever_enabled_multiple": false 2020-03-11T22:36:43.910 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.910 INFO:tasks.workunit.client.0.smithi155.stdout: "standbys": [], 2020-03-11T22:36:43.910 INFO:tasks.workunit.client.0.smithi155.stdout: "filesystems": [] 2020-03-11T22:36:43.910 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.911 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:36:43.911 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:36:43.911 INFO:tasks.workunit.client.0.smithi155.stdout: "auth": { 2020-03-11T22:36:43.911 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:36:43.911 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 5, 2020-03-11T22:36:43.911 INFO:tasks.workunit.client.0.smithi155.stdout: "num_secrets": 3 2020-03-11T22:36:43.911 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.911 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 1, 2020-03-11T22:36:43.912 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_active": 1, 2020-03-11T22:36:43.912 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_unknown": 0, 2020-03-11T22:36:43.912 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osd": 3, 2020-03-11T22:36:43.912 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_sum": { 2020-03-11T22:36:43.912 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:36:43.912 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 48300, 2020-03-11T22:36:43.912 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 100, 2020-03-11T22:36:43.912 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2020-03-11T22:36:43.913 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 300, 2020-03-11T22:36:43.913 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:36:43.913 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:36:43.913 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 100, 2020-03-11T22:36:43.913 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:36:43.913 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:36:43.913 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 100, 2020-03-11T22:36:43.913 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:36:43.914 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:36:43.914 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:36:43.914 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 100, 2020-03-11T22:36:43.914 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 100, 2020-03-11T22:36:43.914 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:36:43.914 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:36:43.914 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:36:43.915 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:36:43.915 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:36:43.915 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:36:43.915 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:36:43.915 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:36:43.915 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:36:43.915 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:36:43.915 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:36:43.916 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:36:43.916 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:36:43.916 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:36:43.916 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:36:43.916 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:36:43.916 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:36:43.916 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:36:43.916 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:36:43.917 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:36:43.917 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:36:43.917 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:36:43.917 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:36:43.917 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:36:43.917 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:36:43.917 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.917 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:36:43.918 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:36:43.918 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:36:43.918 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:36:43.918 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 0, 2020-03-11T22:36:43.918 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 0, 2020-03-11T22:36:43.918 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:36:43.918 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:36:43.918 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:36:43.919 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:36:43.919 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:36:43.919 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.919 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 16, 2020-03-11T22:36:43.919 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 16, 2020-03-11T22:36:43.919 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 2, 2020-03-11T22:36:43.919 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 2, 2020-03-11T22:36:43.920 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 0 2020-03-11T22:36:43.920 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.920 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum": { 2020-03-11T22:36:43.920 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:36:43.920 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:36:43.920 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 3, 2020-03-11T22:36:43.920 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:36:43.920 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:36:43.921 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:36:43.921 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:36:43.921 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3178880, 2020-03-11T22:36:43.921 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 32896, 2020-03-11T22:36:43.921 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 14, 2020-03-11T22:36:43.921 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145713, 2020-03-11T22:36:43.922 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311393920, 2020-03-11T22:36:43.922 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:36:43.922 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:36:43.922 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318867374080, 2020-03-11T22:36:43.922 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:36:43.922 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 33685504, 2020-03-11T22:36:43.922 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 398171, 2020-03-11T22:36:43.922 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:36:43.923 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:36:43.923 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:36:43.923 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 14398, 2020-03-11T22:36:43.923 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221211074 2020-03-11T22:36:43.923 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.923 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:36:43.923 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:36:43.924 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:36:43.924 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 1, 2020-03-11T22:36:43.924 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:36:43.924 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:36:43.924 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:36:43.924 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.924 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:36:43.924 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.925 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum_by_class": { 2020-03-11T22:36:43.926 INFO:tasks.workunit.client.0.smithi155.stdout: "hdd": { 2020-03-11T22:36:43.926 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:36:43.926 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:36:43.926 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 3, 2020-03-11T22:36:43.926 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:36:43.926 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:36:43.926 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:36:43.927 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:36:43.927 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3178880, 2020-03-11T22:36:43.927 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 32896, 2020-03-11T22:36:43.927 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 14, 2020-03-11T22:36:43.927 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145713, 2020-03-11T22:36:43.927 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311393920, 2020-03-11T22:36:43.927 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:36:43.928 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:36:43.928 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318867374080, 2020-03-11T22:36:43.928 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:36:43.928 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 33685504, 2020-03-11T22:36:43.928 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 398171, 2020-03-11T22:36:43.928 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:36:43.928 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:36:43.929 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:36:43.929 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 14398, 2020-03-11T22:36:43.929 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221211074 2020-03-11T22:36:43.929 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.929 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:36:43.929 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:36:43.929 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:36:43.930 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 1, 2020-03-11T22:36:43.930 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:36:43.930 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:36:43.930 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:36:43.930 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.930 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:36:43.930 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:36:43.930 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:36:43.931 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:36:43.931 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:36:43.931 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.931 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:36:43.931 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:36:43.931 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.931 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.932 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_stats": [ 2020-03-11T22:36:43.932 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.932 INFO:tasks.workunit.client.0.smithi155.stdout: "poolid": 1, 2020-03-11T22:36:43.932 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 1, 2020-03-11T22:36:43.932 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:36:43.932 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 48300, 2020-03-11T22:36:43.932 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 100, 2020-03-11T22:36:43.932 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2020-03-11T22:36:43.933 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 300, 2020-03-11T22:36:43.933 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:36:43.933 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:36:43.933 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 100, 2020-03-11T22:36:43.933 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:36:43.933 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:36:43.933 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 100, 2020-03-11T22:36:43.934 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:36:43.934 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:36:43.934 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:36:43.934 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 100, 2020-03-11T22:36:43.934 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 100, 2020-03-11T22:36:43.934 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:36:43.934 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:36:43.934 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:36:43.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:36:43.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:36:43.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:36:43.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:36:43.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:36:43.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:36:43.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:36:43.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:36:43.936 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:36:43.936 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:36:43.936 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:36:43.936 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:36:43.936 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:36:43.936 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:36:43.936 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:36:43.936 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:36:43.937 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:36:43.937 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:36:43.937 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:36:43.937 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:36:43.937 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:36:43.937 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:36:43.937 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.938 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:36:43.938 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:36:43.938 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:36:43.938 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:36:43.938 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 19660800, 2020-03-11T22:36:43.938 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 144900, 2020-03-11T22:36:43.938 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:36:43.938 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:36:43.939 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:36:43.939 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:36:43.939 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:36:43.939 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.939 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 16, 2020-03-11T22:36:43.939 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 16, 2020-03-11T22:36:43.939 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 2, 2020-03-11T22:36:43.939 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 2, 2020-03-11T22:36:43.940 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 3 2020-03-11T22:36:43.940 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.940 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.940 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_stats": [ 2020-03-11T22:36:43.940 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.940 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:36:43.940 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 146028888074 2020-03-11T22:36:43.940 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.941 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.941 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:36:43.941 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 42949672976 2020-03-11T22:36:43.941 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.941 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.941 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:36:43.941 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 150323855370 2020-03-11T22:36:43.942 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.942 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.942 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_state": [ 2020-03-11T22:36:43.942 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.942 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "active+undersized+degraded", 2020-03-11T22:36:43.942 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 1 2020-03-11T22:36:43.942 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.942 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.943 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_osd": [ 2020-03-11T22:36:43.943 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.943 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:36:43.943 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:36:43.943 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 1, 2020-03-11T22:36:43.943 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:36:43.943 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.944 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.944 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:36:43.944 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:36:43.944 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 0, 2020-03-11T22:36:43.944 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:36:43.944 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.944 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.944 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:36:43.945 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 1, 2020-03-11T22:36:43.945 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 1, 2020-03-11T22:36:43.945 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:36:43.945 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.945 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.945 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": [ 2020-03-11T22:36:43.945 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:36:43.945 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:36:43.946 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": {} 2020-03-11T22:36:43.946 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.946 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:36:43.946 INFO:tasks.workunit.client.0.smithi155.stdout: "servicemap": { 2020-03-11T22:36:43.946 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:36:43.946 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:35:05.814651+0000", 2020-03-11T22:36:43.946 INFO:tasks.workunit.client.0.smithi155.stdout: "services": {} 2020-03-11T22:36:43.946 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:36:43.947 INFO:tasks.workunit.client.0.smithi155.stdout: "paxos": { 2020-03-11T22:36:43.947 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:36:43.947 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 300, 2020-03-11T22:36:43.947 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pn": 0, 2020-03-11T22:36:43.947 INFO:tasks.workunit.client.0.smithi155.stdout: "accepted_pn": 0 2020-03-11T22:36:43.947 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:36:43.947 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:36:43.948 INFO:tasks.workunit.client.0.smithi155.stderr:report 3934122019 2020-03-11T22:36:43.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:36:43.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: expr 2 + 2 2020-03-11T22:36:43.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:153: TEST_divergent: objname=existing_4 2020-03-11T22:36:43.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:154: TEST_divergent: echo 'writing non-divergent object existing_4' 2020-03-11T22:36:43.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:155: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:43.949 INFO:tasks.workunit.client.0.smithi155.stdout:writing non-divergent object existing_4 2020-03-11T22:36:44.129 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:44.129 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:36:07.779261+0000 28'100 38:120 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:36:44.129 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:36:44.129 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:44.130 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:36:44.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:156: TEST_divergent: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:36:44.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:159: TEST_divergent: echo 'delay recovery' 2020-03-11T22:36:44.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:160: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:44.252 INFO:tasks.workunit.client.0.smithi155.stdout:delay recovery 2020-03-11T22:36:44.572 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:44.573 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:36:07.779261+0000 28'100 38:120 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:36:44.573 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:36:44.573 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:44.574 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:36:44.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:44.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.0 2020-03-11T22:36:44.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:36:44.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:36:44.583 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:44.583 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:44.584 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:36:44.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.0.asok 2020-03-11T22:36:44.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:36:44.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:36:44.677 INFO:tasks.workunit.client.0.smithi155.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:36:44.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:161: TEST_divergent: for i in $non_divergent 2020-03-11T22:36:44.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: get_asok_path osd.2 2020-03-11T22:36:44.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:36:44.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:36:44.685 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:44.685 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:44.685 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:36:44.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.2.asok 2020-03-11T22:36:44.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: CEPH_ARGS= 2020-03-11T22:36:44.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:163: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:36:44.776 INFO:tasks.workunit.client.0.smithi155.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:36:44.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:167: TEST_divergent: echo 'revive divergent 1' 2020-03-11T22:36:44.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:168: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:44.783 INFO:tasks.workunit.client.0.smithi155.stdout:revive divergent 1 2020-03-11T22:36:45.102 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:45.102 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:36:07.779261+0000 28'100 38:120 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:36:45.102 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:36:45.102 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:45.103 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:36:45.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:169: TEST_divergent: ceph osd set noup 2020-03-11T22:36:45.812 INFO:tasks.workunit.client.0.smithi155.stderr:noup is set 2020-03-11T22:36:45.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:170: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:36:45.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:36:45.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:36:45.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:36:45.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:36:45.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:36:45.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:36:45.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:36:45.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:36:45.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:36:45.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:36:45.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:36:45.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:36:45.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:36:45.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:36:45.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:36:45.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:36:45.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:36:45.826 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:36:45.826 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:45.826 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:36:45.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:36:45.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:36:45.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:36:45.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:36:45.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:36:45.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:36:45.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:36:45.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:45.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:36:45.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:36:45.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:36:45.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:36:45.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:36:45.828 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:36:45.828 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:36:45.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:36:45.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:36:45.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:36:45.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:36:45.849 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:45.847+0000 7f6be2b58ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:45.851 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:45.849+0000 7f6be2b58ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:45.853 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:45.851+0000 7f6be2b58ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:36:46.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:36:46.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:171: TEST_divergent: sleep 5 2020-03-11T22:36:46.369 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:46.366+0000 7f6be2b58ec0 -1 Falling back to public interface 2020-03-11T22:36:47.203 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:36:47.200+0000 7f6be2b58ec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-11T22:36:51.175 INFO:tasks.workunit.client.0.smithi155.stdout:delay recovery divergent 2020-03-11T22:36:51.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:173: TEST_divergent: echo 'delay recovery divergent' 2020-03-11T22:36:51.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:174: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:51.505 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:51.505 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:36:07.779261+0000 38'101 38:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:36:51.505 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:36:51.505 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:51.506 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:36:51.515 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: get_asok_path osd.1 2020-03-11T22:36:51.515 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:36:51.515 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:36:51.515 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:36:51.516 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:36:51.516 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:36:51.516 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.1.asok 2020-03-11T22:36:51.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: CEPH_ARGS= 2020-03-11T22:36:51.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:175: TEST_divergent: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:36:51.606 INFO:tasks.workunit.client.0.smithi155.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:36:51.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:177: TEST_divergent: ceph osd unset noup 2020-03-11T22:36:52.384 INFO:tasks.workunit.client.0.smithi155.stderr:noup is unset 2020-03-11T22:36:52.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:179: TEST_divergent: wait_for_osd up 0 2020-03-11T22:36:52.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:52.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:36:52.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:52.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:52.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:52.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:52.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:52.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:36:52.394 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:36:52.782 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 34 up_thru 37 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/22562,v1:127.0.0.1:6803/22562] [v2:127.0.0.1:6804/22562,v1:127.0.0.1:6805/22562] exists,up e73ac5a4-38f7-49d2-9ebe-aa4155e118b1 2020-03-11T22:36:52.782 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:36:52.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:52.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:52.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:52.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:180: TEST_divergent: wait_for_osd up 1 2020-03-11T22:36:52.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:52.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:36:52.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:52.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:52.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:52.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:52.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:52.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:36:53.125 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/24251,v1:127.0.0.1:6819/24251] [v2:127.0.0.1:6820/24251,v1:127.0.0.1:6821/24251] exists,up 748827f2-8dc8-4162-b402-942b80b4efe3 2020-03-11T22:36:53.125 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:36:53.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:53.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:53.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:53.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:181: TEST_divergent: wait_for_osd up 2 2020-03-11T22:36:53.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:36:53.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:36:53.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:36:53.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:36:53.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:36:53.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:36:53.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:36:53.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:36:53.505 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [15,28) [v2:127.0.0.1:6810/22756,v1:127.0.0.1:6811/22756] [v2:127.0.0.1:6812/22756,v1:127.0.0.1:6813/22756] exists,up ea275d10-e3f1-4218-8e10-8f2a58b11e5b 2020-03-11T22:36:53.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:36:53.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:36:53.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:36:53.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:183: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:53.832 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:53.833 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:36:07.779261+0000 38'101 38:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:36:53.833 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:36:53.833 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:53.834 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:36:53.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:184: TEST_divergent: echo 'wait for peering' 2020-03-11T22:36:53.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:185: TEST_divergent: ceph pg dump pgs 2020-03-11T22:36:53.843 INFO:tasks.workunit.client.0.smithi155.stdout:wait for peering 2020-03-11T22:36:54.163 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:36:54.164 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:36:07.779261+0000 38'101 38:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:36:54.164 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:36:54.164 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:36:54.165 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:36:54.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:186: TEST_divergent: rados -p test put foo /etc/fstab 2020-03-11T22:37:03.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:188: TEST_divergent: echo 'killing divergent 1' 2020-03-11T22:37:03.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:189: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:03.812 INFO:tasks.workunit.client.0.smithi155.stdout:killing divergent 1 2020-03-11T22:37:04.139 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:04.139 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:36:52.701728+0000 38'101 44:134 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:37:04.140 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:37:04.140 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:04.141 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:37:04.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:190: TEST_divergent: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:37:04.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:04.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:04.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:04.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:04.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:04.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:04.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:192: TEST_divergent: echo 'reviving divergent 1' 2020-03-11T22:37:04.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:193: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:04.255 INFO:tasks.workunit.client.0.smithi155.stdout:reviving divergent 1 2020-03-11T22:37:04.575 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:04.576 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 2 2 0 0 47947 0 0 17 17 active+recovering+degraded+wait 2020-03-11T22:36:52.701728+0000 38'101 44:134 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:37:04.576 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:37:04.576 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:04.577 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:37:04.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:194: TEST_divergent: activate_osd td/divergent-priors 1 2020-03-11T22:37:04.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:37:04.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:37:04.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:37:04.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:37:04.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:37:04.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:04.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:04.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:04.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:04.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:37:04.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:37:04.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:04.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:37:04.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:04.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:37:04.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:04.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:04.593 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:04.593 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:04.593 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:37:04.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:37:04.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:37:04.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:04.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:04.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:04.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:04.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:04.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:04.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:37:04.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:37:04.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:37:04.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:37:04.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:04.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:37:04.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:37:04.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:37:04.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:37:04.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:37:04.596 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:37:04.614 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:04.610+0000 7fd5c0086ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:04.615 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:04.612+0000 7fd5c0086ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:04.618 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:04.614+0000 7fd5c0086ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:04.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:37:04.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:04.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:37:04.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:04.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:04.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:04.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:04.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:04.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:37:04.937 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:37:05.135 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:05.131+0000 7fd5c0086ec0 -1 Falling back to public interface 2020-03-11T22:37:05.283 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 42 up_thru 42 down_at 30 last_clean_interval [10,28) [v2:127.0.0.1:6818/24251,v1:127.0.0.1:6819/24251] [v2:127.0.0.1:6820/24251,v1:127.0.0.1:6821/24251] exists,up 748827f2-8dc8-4162-b402-942b80b4efe3 2020-03-11T22:37:05.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:05.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:05.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:05.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:196: TEST_divergent: sleep 20 2020-03-11T22:37:06.025 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:06.021+0000 7fd5c0086ec0 -1 osd.1 44 log_to_monitors {default=true} 2020-03-11T22:37:25.284 INFO:tasks.workunit.client.0.smithi155.stdout:allowing recovery 2020-03-11T22:37:25.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:198: TEST_divergent: echo 'allowing recovery' 2020-03-11T22:37:25.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:199: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:25.612 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:25.612 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:37:07.052258+0000 44'102 47:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:37:25.612 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:37:25.613 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:25.614 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:37:25.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:37:25.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:37:25.743 INFO:tasks.workunit.client.0.smithi155.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:37:25.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:37:25.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:37:25.871 INFO:tasks.workunit.client.0.smithi155.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:37:25.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:201: TEST_divergent: for i in $osds 2020-03-11T22:37:25.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:203: TEST_divergent: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:37:25.999 INFO:tasks.workunit.client.0.smithi155.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:37:26.007 INFO:tasks.workunit.client.0.smithi155.stdout:reading divergent objects 2020-03-11T22:37:26.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:206: TEST_divergent: echo 'reading divergent objects' 2020-03-11T22:37:26.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:207: TEST_divergent: ceph pg dump pgs 2020-03-11T22:37:26.325 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:37:26.325 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 0 0 0 0 48430 0 0 18 18 active+clean 2020-03-11T22:37:07.052258+0000 44'102 47:151 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:35:26.861047+0000 0'0 2020-03-11T22:35:26.861047+0000 0 2020-03-11T22:37:26.325 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:37:26.326 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:37:26.327 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:37:26.335 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: expr 2 + 2 2020-03-11T22:37:26.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: seq 1 4 2020-03-11T22:37:26.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:37:26.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:37:26.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:37:26.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:37:26.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:37:26.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:37:26.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:208: TEST_divergent: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:37:26.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:210: TEST_divergent: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:37:26.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:212: TEST_divergent: rm -f td/divergent-priors/existing 2020-03-11T22:37:26.500 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:37:26.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:214: TEST_divergent: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:37:26.504 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:36:52.663+0000 7f6bc0d13700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4522.0:1 2020-03-11T22:35:48.553313+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:37:26.504 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:36:52.663+0000 7f6bc0d13700 20 _merge_object_divergent_entries: keeping 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4522.0:1 2020-03-11T22:35:48.553313+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:37:26.505 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:36:52.663+0000 7f6bc0d13700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:37:26.505 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:36:52.663+0000 7f6bc0d13700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 28'100 first_divergent_update: 28'101 last_divergent_update: 28'101 2020-03-11T22:37:26.505 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:36:52.663+0000 7f6bc0d13700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:37:26.505 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:36:52.663+0000 7f6bc0d13700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:37:26.505 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:36:52.663+0000 7f6bc0d13700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'101 2020-03-11T22:37:26.505 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:36:52.663+0000 7f6bc0d13700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 28'101 (28'100) modify 1:c156033e:::existing_100:head by client.4522.0:1 2020-03-11T22:35:48.553313+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:37:26.505 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:36:52.664+0000 7f6bc0d13700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:37:26.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: find td/divergent-priors -name '*osd*log' 2020-03-11T22:37:26.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:216: TEST_divergent: grep -q '_merge_object_divergent_entries.*cannot roll back, removing and adding to missing' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:37:26.509 INFO:tasks.workunit.client.0.smithi155.stdout:success 2020-03-11T22:37:26.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:221: TEST_divergent: echo success 2020-03-11T22:37:26.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:223: TEST_divergent: delete_pool test 2020-03-11T22:37:26.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:37:26.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:37:26.976 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test' does not exist 2020-03-11T22:37:26.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:224: TEST_divergent: kill_daemons td/divergent-priors 2020-03-11T22:37:26.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:26.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:26.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:26.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:26.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:27.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:27.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:37:27.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:37:27.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:37:27.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:37:27.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:27.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:27.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:27.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:27.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:27.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:27.097 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:37:27.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:37:27.098 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:37:27.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:37:27.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:37:27.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:37:27.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:37:27.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:37:27.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:37:27.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:37:27.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:37:27.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:37:27.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:37:27.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:37:27.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:27.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:37:27.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17464 2020-03-11T22:37:27.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:37:27.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:37:27.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:37:27.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:37:27.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:37:27.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:37:27.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:37:27.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:37:27.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:37:27.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:37:27.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:37:27.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:37:27.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:37:27.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:37:27.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:37:27.118 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:37:27.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:37:27.118 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:37:27.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:37:27.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:37:27.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:37:27.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:37:27.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:37:27.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:37:27.121 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:37:27.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:37:27.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:37:27.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:37:27.123 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:37:27.123 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:27.123 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:37:27.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17464 2020-03-11T22:37:27.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:37:27.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:37:27.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:37:27.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:37:27.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:27.126 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:37:27.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17464 2020-03-11T22:37:27.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:37:27.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:37:27.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:37:27.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:37:27.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_2 td/divergent-priors 2020-03-11T22:37:27.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:421: TEST_divergent_2: local dir=td/divergent-priors 2020-03-11T22:37:27.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:424: TEST_divergent_2: local dummyfile=/etc/fstab 2020-03-11T22:37:27.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:425: TEST_divergent_2: local dummyfile2=/etc/resolv.conf 2020-03-11T22:37:27.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:427: TEST_divergent_2: local num_osds=3 2020-03-11T22:37:27.128 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: expr 3 - 1 2020-03-11T22:37:27.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: seq 0 2 2020-03-11T22:37:27.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:428: TEST_divergent_2: local 'osds=0 2020-03-11T22:37:27.130 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:37:27.130 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:37:27.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:429: TEST_divergent_2: run_mon td/divergent-priors a 2020-03-11T22:37:27.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:37:27.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:37:27.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:37:27.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:37:27.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:37:27.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:37:27.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:37:27.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:27.466 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:27.466 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:27.467 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:27.467 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:37:27.467 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:37:27.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:37:27.617 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:37:27.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:37:27.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:37:27.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:37:27.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:37:27.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:37:27.619 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:37:27.619 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:37:27.620 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:37:27.620 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:27.620 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:27.620 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:37:27.620 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-mon.a.asok 2020-03-11T22:37:27.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:37:27.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17464/ceph-mon.a.asok config get fsid 2020-03-11T22:37:27.706 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:37:27.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:37:27.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:37:27.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:37:27.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:37:27.707 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:37:27.707 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:37:27.708 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:37:27.708 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:37:27.708 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:27.708 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:37:27.708 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-mon.a.asok 2020-03-11T22:37:27.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:37:27.709 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17464/ceph-mon.a.asok config get mon_host 2020-03-11T22:37:27.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:430: TEST_divergent_2: run_mgr td/divergent-priors x 2020-03-11T22:37:27.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:37:27.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:37:27.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:37:27.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:37:27.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:37:27.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:37:28.699 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:37:28.699 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:28.700 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:28.700 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:28.700 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:28.700 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:37:28.700 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:37:28.701 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:37:28.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:37:28.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:37:28.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 0 2020-03-11T22:37:28.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:37:28.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:37:28.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:37:28.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:37:28.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:37:28.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:28.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:28.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:28.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:28.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:37:28.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:37:28.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:28.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:37:28.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:28.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:37:28.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:28.736 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:28.736 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:28.736 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:28.737 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:37:28.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:37:28.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:37:28.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:28.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:37:28.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:37:28.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:28.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:28.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:28.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:28.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:28.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:37:28.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:37:28.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:37:28.739 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:37:28.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=9c7f1786-6f01-4f0d-8a2e-5034ec430e9d 2020-03-11T22:37:28.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 9c7f1786-6f01-4f0d-8a2e-5034ec430e9d' 2020-03-11T22:37:28.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:37:28.740 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 9c7f1786-6f01-4f0d-8a2e-5034ec430e9d 2020-03-11T22:37:28.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAoaGleUYVoLBAAcMlvju3SAX/aWMoybnd4AQ== 2020-03-11T22:37:28.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAoaGleUYVoLBAAcMlvju3SAX/aWMoybnd4AQ=="}' 2020-03-11T22:37:28.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 9c7f1786-6f01-4f0d-8a2e-5034ec430e9d -i td/divergent-priors/0/new.json 2020-03-11T22:37:29.056 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:37:29.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:37:29.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAoaGleUYVoLBAAcMlvju3SAX/aWMoybnd4AQ== --osd-uuid 9c7f1786-6f01-4f0d-8a2e-5034ec430e9d 2020-03-11T22:37:29.095 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:29.093+0000 7ff0d261aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:29.097 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:29.094+0000 7ff0d261aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:29.098 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:29.095+0000 7ff0d261aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:29.098 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:29.096+0000 7ff0d261aec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:37:29.098 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:29.096+0000 7ff0d261aec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:37:29.098 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:29.096+0000 7ff0d261aec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:37:29.141 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:29.139+0000 7ff0d261aec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:37:32.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:37:32.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:37:32.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:37:32.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:37:32.139 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2020-03-11T22:37:32.586 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:37:32.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:37:32.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:32.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:37:32.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:37:32.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:37:32.612 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:32.610+0000 7f8945aa7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:32.614 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:32.612+0000 7f8945aa7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:32.616 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:32.614+0000 7f8945aa7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:32.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:37:32.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:32.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:37:32.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:32.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:32.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:32.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:32.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:32.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:32.935 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:37:33.132 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:33.130+0000 7f8945aa7ec0 -1 Falling back to public interface 2020-03-11T22:37:33.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:33.867 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:33.864+0000 7f8945aa7ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:37:34.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:34.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:34.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:34.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:34.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:37:34.298 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:37:34.650 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/25553,v1:127.0.0.1:6803/25553] [v2:127.0.0.1:6804/25553,v1:127.0.0.1:6805/25553] exists,up 9c7f1786-6f01-4f0d-8a2e-5034ec430e9d 2020-03-11T22:37:34.651 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 1b163036-a7f8-46e5-b80e-ffab673e9327 2020-03-11T22:37:34.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:34.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:34.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:34.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:37:34.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 1 2020-03-11T22:37:34.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:37:34.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:37:34.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:37:34.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:37:34.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:37:34.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:34.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:34.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:34.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:34.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:37:34.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:37:34.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:34.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:37:34.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:34.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:37:34.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:34.655 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:34.655 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:34.655 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:34.655 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:37:34.655 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:37:34.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:37:34.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:34.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:37:34.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:37:34.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:34.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:34.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:34.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:34.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:34.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:37:34.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:37:34.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:37:34.657 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:37:34.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=1b163036-a7f8-46e5-b80e-ffab673e9327 2020-03-11T22:37:34.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 1b163036-a7f8-46e5-b80e-ffab673e9327' 2020-03-11T22:37:34.657 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:37:34.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQAuaGleGsu1JxAA4AtOJJPM4zoA4kDILfUzVw== 2020-03-11T22:37:34.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQAuaGleGsu1JxAA4AtOJJPM4zoA4kDILfUzVw=="}' 2020-03-11T22:37:34.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 1b163036-a7f8-46e5-b80e-ffab673e9327 -i td/divergent-priors/1/new.json 2020-03-11T22:37:35.154 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:37:35.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:37:35.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQAuaGleGsu1JxAA4AtOJJPM4zoA4kDILfUzVw== --osd-uuid 1b163036-a7f8-46e5-b80e-ffab673e9327 2020-03-11T22:37:35.192 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:35.190+0000 7f5bc65f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:35.194 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:35.192+0000 7f5bc65f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:35.196 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:35.193+0000 7f5bc65f9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:35.196 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:35.194+0000 7f5bc65f9ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:37:35.197 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:35.194+0000 7f5bc65f9ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:37:35.197 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:35.194+0000 7f5bc65f9ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:37:35.282 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:35.280+0000 7f5bc65f9ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:37:38.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:37:38.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:37:38.308 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2020-03-11T22:37:38.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:37:38.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:37:38.762 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:37:38.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:37:38.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:38.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:37:38.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:37:38.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:37:38.788 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:38.785+0000 7f75a4933ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:38.790 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:38.787+0000 7f75a4933ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:38.791 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:38.788+0000 7f75a4933ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:39.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:37:39.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:39.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:37:39.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:39.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:39.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:39.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:39.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:39.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:37:39.111 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:37:39.305 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:39.302+0000 7f75a4933ec0 -1 Falling back to public interface 2020-03-11T22:37:39.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:40.122 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:40.120+0000 7f75a4933ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:37:40.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:40.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:40.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:40.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:40.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:37:40.458 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:37:40.819 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/25908,v1:127.0.0.1:6811/25908] [v2:127.0.0.1:6812/25908,v1:127.0.0.1:6813/25908] exists,up 1b163036-a7f8-46e5-b80e-ffab673e9327 2020-03-11T22:37:40.819 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 9b1fa1a6-0529-400f-aeb3-4b08986f4d72 2020-03-11T22:37:40.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:40.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:40.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:40.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:431: TEST_divergent_2: for i in $osds 2020-03-11T22:37:40.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:433: TEST_divergent_2: run_osd td/divergent-priors 2 2020-03-11T22:37:40.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:37:40.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:37:40.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:37:40.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:37:40.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:37:40.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:37:40.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:37:40.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:37:40.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:37:40.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:37:40.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:37:40.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:37:40.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:37:40.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:37:40.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:37:40.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:37:40.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:37:40.823 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:37:40.823 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:37:40.823 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:37:40.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:37:40.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:37:40.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:37:40.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:37:40.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:37:40.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:37:40.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:37:40.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:37:40.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:37:40.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:40.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:37:40.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:37:40.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:37:40.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:37:40.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=9b1fa1a6-0529-400f-aeb3-4b08986f4d72 2020-03-11T22:37:40.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 9b1fa1a6-0529-400f-aeb3-4b08986f4d72' 2020-03-11T22:37:40.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:37:40.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA0aGleUK+tMRAAfWt8hkposZ/WDpfECZw5eQ== 2020-03-11T22:37:40.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA0aGleUK+tMRAAfWt8hkposZ/WDpfECZw5eQ=="}' 2020-03-11T22:37:40.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 9b1fa1a6-0529-400f-aeb3-4b08986f4d72 -i td/divergent-priors/2/new.json 2020-03-11T22:37:41.339 INFO:tasks.workunit.client.0.smithi155.stdout:2 2020-03-11T22:37:41.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:37:41.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA0aGleUK+tMRAAfWt8hkposZ/WDpfECZw5eQ== --osd-uuid 9b1fa1a6-0529-400f-aeb3-4b08986f4d72 2020-03-11T22:37:41.378 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:41.375+0000 7f73e6983ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:41.397 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:41.394+0000 7f73e6983ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:41.399 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:41.396+0000 7f73e6983ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:41.400 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:41.397+0000 7f73e6983ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:37:41.400 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:41.397+0000 7f73e6983ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:37:41.400 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:41.397+0000 7f73e6983ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:37:41.463 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:41.461+0000 7f73e6983ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:37:44.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:37:44.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:37:44.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:37:44.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:37:44.467 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2020-03-11T22:37:44.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:37:44.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:37:44.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:37:44.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:37:44.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:37:44.925 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:37:44.950 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:44.948+0000 7f417aa6cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:44.952 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:44.950+0000 7f417aa6cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:44.954 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:44.952+0000 7f417aa6cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:37:45.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:37:45.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:37:45.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:37:45.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:37:45.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:37:45.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:45.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:37:45.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:45.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:45.275 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:37:45.470 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:45.467+0000 7f417aa6cec0 -1 Falling back to public interface 2020-03-11T22:37:45.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:46.305 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:37:46.302+0000 7f417aa6cec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:37:46.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:46.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:46.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:37:46.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:46.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:46.626 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:37:46.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:37:47.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:37:47.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:37:47.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:37:47.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:37:47.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:37:47.976 INFO:tasks.workunit.client.0.smithi155.stdout:2 2020-03-11T22:37:48.331 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/26263,v1:127.0.0.1:6819/26263] [v2:127.0.0.1:6820/26263,v1:127.0.0.1:6821/26263] exists,up 9b1fa1a6-0529-400f-aeb3-4b08986f4d72 2020-03-11T22:37:48.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:37:48.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:37:48.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:37:48.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:436: TEST_divergent_2: ceph osd set noout 2020-03-11T22:37:48.925 INFO:tasks.workunit.client.0.smithi155.stderr:noout is set 2020-03-11T22:37:48.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:437: TEST_divergent_2: ceph osd set noin 2020-03-11T22:37:49.660 INFO:tasks.workunit.client.0.smithi155.stderr:noin is set 2020-03-11T22:37:49.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:438: TEST_divergent_2: ceph osd set nodown 2020-03-11T22:37:50.429 INFO:tasks.workunit.client.0.smithi155.stderr:nodown is set 2020-03-11T22:37:50.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:439: TEST_divergent_2: create_pool test 1 1 2020-03-11T22:37:50.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:37:51.038 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test' already exists 2020-03-11T22:37:51.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:37:52.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:440: TEST_divergent_2: ceph osd pool set test size 3 2020-03-11T22:37:52.709 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 size to 3 2020-03-11T22:37:52.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:441: TEST_divergent_2: ceph osd pool set test min_size 2 2020-03-11T22:37:53.418 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 min_size to 2 2020-03-11T22:37:53.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:443: TEST_divergent_2: flush_pg_stats 2020-03-11T22:37:53.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:37:53.426 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:37:53.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:37:53.870 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:37:53.870 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:37:53.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:37:53.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:53.871 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:37:53.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836487 2020-03-11T22:37:53.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487' 2020-03-11T22:37:53.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:53.940 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:37:54.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705669 2020-03-11T22:37:54.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-38654705669' 2020-03-11T22:37:54.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:54.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:37:54.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542148 2020-03-11T22:37:54.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-38654705669 2-60129542148' 2020-03-11T22:37:54.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:54.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836487 2020-03-11T22:37:54.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:54.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:37:54.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836487 2020-03-11T22:37:54.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:54.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836487 2020-03-11T22:37:54.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-03-11T22:37:54.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:54.224 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836487 2020-03-11T22:37:54.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836487 2020-03-11T22:37:54.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:54.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705669 2020-03-11T22:37:54.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:54.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:37:54.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705669 2020-03-11T22:37:54.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:54.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705669 2020-03-11T22:37:54.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2020-03-11T22:37:54.564 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705669 2020-03-11T22:37:54.565 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:37:54.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705667 -lt 38654705669 2020-03-11T22:37:54.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:37:55.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:37:55.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:37:56.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705669 -lt 38654705669 2020-03-11T22:37:56.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:56.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542148 2020-03-11T22:37:56.265 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:56.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:37:56.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542148 2020-03-11T22:37:56.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:56.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542148 2020-03-11T22:37:56.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2020-03-11T22:37:56.268 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542148 2020-03-11T22:37:56.268 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:37:56.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542148 -lt 60129542148 2020-03-11T22:37:56.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:444: TEST_divergent_2: wait_for_clean 2020-03-11T22:37:56.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:37:56.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:37:56.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:37:56.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:37:56.638 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:37:56.639 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:37:56.639 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:37:56.639 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:37:56.639 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:37:56.640 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:37:56.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:37:56.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:37:56.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:37:56.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:37:56.731 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:37:57.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:37:57.079 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:37:57.079 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:37:57.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:37:57.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:57.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:37:57.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:37:57.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:37:57.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:57.208 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:37:57.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705672 2020-03-11T22:37:57.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-38654705672' 2020-03-11T22:37:57.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:37:57.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:37:57.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542150 2020-03-11T22:37:57.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-38654705672 2-60129542150' 2020-03-11T22:37:57.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:57.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:37:57.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:57.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:37:57.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:37:57.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:57.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:37:57.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:37:57.470 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:57.471 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:37:57.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836488 -lt 21474836490 2020-03-11T22:37:57.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:37:58.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:37:58.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:37:59.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836490 2020-03-11T22:37:59.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:59.190 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705672 2020-03-11T22:37:59.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:59.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:37:59.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705672 2020-03-11T22:37:59.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:59.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705672 2020-03-11T22:37:59.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2020-03-11T22:37:59.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:37:59.193 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705672 2020-03-11T22:37:59.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705672 -lt 38654705672 2020-03-11T22:37:59.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:37:59.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542150 2020-03-11T22:37:59.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:37:59.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:37:59.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542150 2020-03-11T22:37:59.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:37:59.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542150 2020-03-11T22:37:59.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2020-03-11T22:37:59.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:37:59.531 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542150 2020-03-11T22:37:59.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542151 -lt 60129542150 2020-03-11T22:37:59.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:37:59.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:37:59.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:00.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:38:00.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:00.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:00.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:00.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:00.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:00.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:00.330 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:00.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:38:00.671 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:00.672 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:00.672 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:01.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:38:01.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:38:01.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:38:01.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: ceph pg dump pgs --format=json 2020-03-11T22:38:01.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: jq '.pg_stats[0].up_primary' 2020-03-11T22:38:01.453 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:38:01.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:447: TEST_divergent_2: local divergent=1 2020-03-11T22:38:01.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:448: TEST_divergent_2: echo 'primary and soon to be divergent is 1' 2020-03-11T22:38:01.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:449: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:01.462 INFO:tasks.workunit.client.0.smithi155.stdout:primary and soon to be divergent is 1 2020-03-11T22:38:01.784 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:01.784 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:37:51.839766+0000 0'0 24:11 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:38:01.784 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:38:01.784 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:01.785 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:38:01.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:450: TEST_divergent_2: local non_divergent= 2020-03-11T22:38:01.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:38:01.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 0 = 1 ']' 2020-03-11T22:38:01.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0' 2020-03-11T22:38:01.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:38:01.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 1 = 1 ']' 2020-03-11T22:38:01.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:454: TEST_divergent_2: continue 2020-03-11T22:38:01.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:451: TEST_divergent_2: for i in $osds 2020-03-11T22:38:01.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:453: TEST_divergent_2: '[' 2 = 1 ']' 2020-03-11T22:38:01.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:456: TEST_divergent_2: non_divergent=' 0 2' 2020-03-11T22:38:01.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:459: TEST_divergent_2: echo 'writing initial objects' 2020-03-11T22:38:01.796 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: seq 1 100 2020-03-11T22:38:01.796 INFO:tasks.workunit.client.0.smithi155.stdout:writing initial objects 2020-03-11T22:38:01.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:01.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_1 /etc/fstab 2020-03-11T22:38:01.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:01.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_2 /etc/fstab 2020-03-11T22:38:01.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:01.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_3 /etc/fstab 2020-03-11T22:38:02.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_4 /etc/fstab 2020-03-11T22:38:02.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_5 /etc/fstab 2020-03-11T22:38:02.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_6 /etc/fstab 2020-03-11T22:38:02.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_7 /etc/fstab 2020-03-11T22:38:02.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_8 /etc/fstab 2020-03-11T22:38:02.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_9 /etc/fstab 2020-03-11T22:38:02.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_10 /etc/fstab 2020-03-11T22:38:02.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_11 /etc/fstab 2020-03-11T22:38:02.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_12 /etc/fstab 2020-03-11T22:38:02.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_13 /etc/fstab 2020-03-11T22:38:02.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_14 /etc/fstab 2020-03-11T22:38:02.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_15 /etc/fstab 2020-03-11T22:38:02.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_16 /etc/fstab 2020-03-11T22:38:02.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_17 /etc/fstab 2020-03-11T22:38:02.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_18 /etc/fstab 2020-03-11T22:38:02.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_19 /etc/fstab 2020-03-11T22:38:02.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:02.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_20 /etc/fstab 2020-03-11T22:38:03.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_21 /etc/fstab 2020-03-11T22:38:03.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_22 /etc/fstab 2020-03-11T22:38:03.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_23 /etc/fstab 2020-03-11T22:38:03.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_24 /etc/fstab 2020-03-11T22:38:03.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_25 /etc/fstab 2020-03-11T22:38:03.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_26 /etc/fstab 2020-03-11T22:38:03.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_27 /etc/fstab 2020-03-11T22:38:03.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_28 /etc/fstab 2020-03-11T22:38:03.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_29 /etc/fstab 2020-03-11T22:38:03.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_30 /etc/fstab 2020-03-11T22:38:03.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_31 /etc/fstab 2020-03-11T22:38:03.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_32 /etc/fstab 2020-03-11T22:38:03.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_33 /etc/fstab 2020-03-11T22:38:03.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_34 /etc/fstab 2020-03-11T22:38:03.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_35 /etc/fstab 2020-03-11T22:38:03.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:03.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_36 /etc/fstab 2020-03-11T22:38:04.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_37 /etc/fstab 2020-03-11T22:38:04.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_38 /etc/fstab 2020-03-11T22:38:04.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_39 /etc/fstab 2020-03-11T22:38:04.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_40 /etc/fstab 2020-03-11T22:38:04.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_41 /etc/fstab 2020-03-11T22:38:04.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_42 /etc/fstab 2020-03-11T22:38:04.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_43 /etc/fstab 2020-03-11T22:38:04.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_44 /etc/fstab 2020-03-11T22:38:04.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_45 /etc/fstab 2020-03-11T22:38:04.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_46 /etc/fstab 2020-03-11T22:38:04.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_47 /etc/fstab 2020-03-11T22:38:04.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_48 /etc/fstab 2020-03-11T22:38:04.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_49 /etc/fstab 2020-03-11T22:38:04.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_50 /etc/fstab 2020-03-11T22:38:04.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:04.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_51 /etc/fstab 2020-03-11T22:38:05.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_52 /etc/fstab 2020-03-11T22:38:05.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_53 /etc/fstab 2020-03-11T22:38:05.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_54 /etc/fstab 2020-03-11T22:38:05.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_55 /etc/fstab 2020-03-11T22:38:05.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_56 /etc/fstab 2020-03-11T22:38:05.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_57 /etc/fstab 2020-03-11T22:38:05.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_58 /etc/fstab 2020-03-11T22:38:05.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_59 /etc/fstab 2020-03-11T22:38:05.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_60 /etc/fstab 2020-03-11T22:38:05.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_61 /etc/fstab 2020-03-11T22:38:05.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_62 /etc/fstab 2020-03-11T22:38:05.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_63 /etc/fstab 2020-03-11T22:38:05.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_64 /etc/fstab 2020-03-11T22:38:05.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_65 /etc/fstab 2020-03-11T22:38:05.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_66 /etc/fstab 2020-03-11T22:38:05.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_67 /etc/fstab 2020-03-11T22:38:05.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:05.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_68 /etc/fstab 2020-03-11T22:38:06.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_69 /etc/fstab 2020-03-11T22:38:06.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_70 /etc/fstab 2020-03-11T22:38:06.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_71 /etc/fstab 2020-03-11T22:38:06.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_72 /etc/fstab 2020-03-11T22:38:06.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_73 /etc/fstab 2020-03-11T22:38:06.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_74 /etc/fstab 2020-03-11T22:38:06.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_75 /etc/fstab 2020-03-11T22:38:06.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_76 /etc/fstab 2020-03-11T22:38:06.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_77 /etc/fstab 2020-03-11T22:38:06.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_78 /etc/fstab 2020-03-11T22:38:06.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_79 /etc/fstab 2020-03-11T22:38:06.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_80 /etc/fstab 2020-03-11T22:38:06.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_81 /etc/fstab 2020-03-11T22:38:06.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_82 /etc/fstab 2020-03-11T22:38:06.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_83 /etc/fstab 2020-03-11T22:38:06.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:06.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_84 /etc/fstab 2020-03-11T22:38:07.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_85 /etc/fstab 2020-03-11T22:38:07.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_86 /etc/fstab 2020-03-11T22:38:07.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_87 /etc/fstab 2020-03-11T22:38:07.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_88 /etc/fstab 2020-03-11T22:38:07.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_89 /etc/fstab 2020-03-11T22:38:07.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_90 /etc/fstab 2020-03-11T22:38:07.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_91 /etc/fstab 2020-03-11T22:38:07.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_92 /etc/fstab 2020-03-11T22:38:07.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_93 /etc/fstab 2020-03-11T22:38:07.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_94 /etc/fstab 2020-03-11T22:38:07.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_95 /etc/fstab 2020-03-11T22:38:07.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_96 /etc/fstab 2020-03-11T22:38:07.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:07.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_97 /etc/fstab 2020-03-11T22:38:08.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:08.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_98 /etc/fstab 2020-03-11T22:38:08.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:08.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_99 /etc/fstab 2020-03-11T22:38:08.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:461: TEST_divergent_2: for i in $(seq 1 $testobjects) 2020-03-11T22:38:08.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:463: TEST_divergent_2: rados -p test put existing_100 /etc/fstab 2020-03-11T22:38:08.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:38:08.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:466: TEST_divergent_2: wait_for_clean 2020-03-11T22:38:08.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:38:08.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:38:08.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:38:08.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:38:08.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:38:08.270 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:38:08.270 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:38:08.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:38:08.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:38:08.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:38:08.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:38:08.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:38:08.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:38:08.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:08.326 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:08.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:08.665 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:38:08.665 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:38:08.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:08.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:08.666 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:08.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836494 2020-03-11T22:38:08.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494' 2020-03-11T22:38:08.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:08.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:08.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705676 2020-03-11T22:38:08.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-38654705676' 2020-03-11T22:38:08.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:08.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:09.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542155 2020-03-11T22:38:09.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836494 1-38654705676 2-60129542155' 2020-03-11T22:38:09.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:09.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836494 2020-03-11T22:38:09.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:09.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:09.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836494 2020-03-11T22:38:09.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:09.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836494 2020-03-11T22:38:09.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836494' 2020-03-11T22:38:09.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:09.057 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836494 2020-03-11T22:38:09.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836494 2020-03-11T22:38:09.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:10.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:10.400 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:10.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836494 2020-03-11T22:38:10.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:10.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705676 2020-03-11T22:38:10.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:10.755 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:10.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705676 2020-03-11T22:38:10.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:10.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705676 2020-03-11T22:38:10.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2020-03-11T22:38:10.757 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:10.757 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705676 2020-03-11T22:38:11.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705676 -lt 38654705676 2020-03-11T22:38:11.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:11.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542155 2020-03-11T22:38:11.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:11.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:11.102 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542155 2020-03-11T22:38:11.102 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:11.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542155 2020-03-11T22:38:11.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2020-03-11T22:38:11.104 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:11.104 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542155 2020-03-11T22:38:11.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542155 -lt 60129542155 2020-03-11T22:38:11.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:38:11.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:11.447 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:11.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:38:11.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:11.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:11.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:11.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:11.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:11.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:11.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:12.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:38:12.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:12.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:12.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:12.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:38:12.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:38:12.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:38:12.670 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: get_pg test existing_1 2020-03-11T22:38:12.670 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:38:12.670 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:38:12.670 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:38:12.670 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:38:13.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:468: TEST_divergent_2: local pgid=1.0 2020-03-11T22:38:13.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:471: TEST_divergent_2: echo 'blackholing osds 0 2' 2020-03-11T22:38:13.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:472: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:13.032 INFO:tasks.workunit.client.0.smithi155.stdout:blackholing osds 0 2 2020-03-11T22:38:13.353 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:13.353 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:37:51.839766+0000 28'100 28:111 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:38:13.354 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:38:13.354 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:13.355 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:38:13.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:13.364 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:38:13.364 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:38:13.364 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:38:13.364 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:13.364 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:13.364 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:38:13.365 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.0.asok 2020-03-11T22:38:13.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:13.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:38:13.450 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:38:13.451 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:38:13.451 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:38:13.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:473: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:13.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:38:13.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:38:13.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:38:13.462 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:38:13.462 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:13.463 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:38:13.463 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.2.asok 2020-03-11T22:38:13.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:38:13.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:475: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:38:13.545 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:38:13.546 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:38:13.546 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:38:13.552 INFO:tasks.workunit.client.0.smithi155.stdout:create new divergent objects 2020-03-11T22:38:13.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:479: TEST_divergent_2: echo 'create new divergent objects' 2020-03-11T22:38:13.553 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: seq 1 2 2020-03-11T22:38:13.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:38:13.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:480: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:38:13.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_1 2020-03-11T22:38:13.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:485: TEST_divergent_2: echo 'writing divergent objects' 2020-03-11T22:38:13.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:482: TEST_divergent_2: rados -p test create newobject_2 2020-03-11T22:38:13.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: seq 1 2 2020-03-11T22:38:13.555 INFO:tasks.workunit.client.0.smithi155.stdout:writing divergent objects 2020-03-11T22:38:13.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:38:13.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:486: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:38:13.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:38:13.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:491: TEST_divergent_2: echo 'remove divergent objects' 2020-03-11T22:38:13.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:488: TEST_divergent_2: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:38:13.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: seq 1 2 2020-03-11T22:38:13.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:38:13.557 INFO:tasks.workunit.client.0.smithi155.stdout:remove divergent objects 2020-03-11T22:38:13.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 1 + 2 2020-03-11T22:38:13.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=3 2020-03-11T22:38:13.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:492: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:38:13.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_3 2020-03-11T22:38:13.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: expr 2 + 2 2020-03-11T22:38:13.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:494: TEST_divergent_2: rmi=4 2020-03-11T22:38:13.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:497: TEST_divergent_2: sleep 10 2020-03-11T22:38:13.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:495: TEST_divergent_2: rados -p test rm existing_4 2020-03-11T22:38:23.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:498: TEST_divergent_2: killall -9 rados 2020-03-11T22:38:23.564 INFO:tasks.workunit.client.0.smithi155.stderr:rados: no process found 2020-03-11T22:38:23.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:501: TEST_divergent_2: echo 'killing all the osds' 2020-03-11T22:38:23.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:502: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:23.564 INFO:tasks.workunit.client.0.smithi155.stdout:killing all the osds 2020-03-11T22:38:23.885 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:23.886 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:37:51.839766+0000 28'106 28:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:38:23.886 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:38:23.886 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:23.887 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:38:23.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:503: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:38:23.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:38:23.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:38:23.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:38:23.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:38:23.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:38:24.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:38:24.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:38:24.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.0 2020-03-11T22:38:24.497 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2020-03-11T22:38:24.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:38:24.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.1 2020-03-11T22:38:24.996 INFO:tasks.workunit.client.0.smithi155.stderr:osd.1 is already down. 2020-03-11T22:38:25.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:504: TEST_divergent_2: for i in $osds 2020-03-11T22:38:25.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:506: TEST_divergent_2: ceph osd down osd.2 2020-03-11T22:38:25.499 INFO:tasks.workunit.client.0.smithi155.stderr:osd.2 is already down. 2020-03-11T22:38:25.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:25.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.0 2020-03-11T22:38:26.005 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already out. 2020-03-11T22:38:26.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:508: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:26.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:510: TEST_divergent_2: ceph osd out osd.2 2020-03-11T22:38:26.540 INFO:tasks.workunit.client.0.smithi155.stderr:osd.2 is already out. 2020-03-11T22:38:26.548 INFO:tasks.workunit.client.0.smithi155.stdout:bringing up non_divergent 0 2 2020-03-11T22:38:26.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:514: TEST_divergent_2: echo 'bringing up non_divergent 0 2' 2020-03-11T22:38:26.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:515: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:38:26.865 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:38:26.866 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:37:51.839766+0000 28'106 28:117 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:38:26.866 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:38:26.866 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:38:26.867 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:38:26.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:26.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 0 2020-03-11T22:38:26.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:26.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:26.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:38:26.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:26.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:38:26.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:26.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:26.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:26.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:26.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:38:26.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:38:26.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:26.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:26.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:26.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:26.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:26.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:26.881 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:26.881 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:26.881 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:38:26.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:38:26.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:38:26.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:26.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:26.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:26.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:26.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:26.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:26.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:26.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:26.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:38:26.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:38:26.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:26.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:38:26.884 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:38:26.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:38:26.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:26.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:26.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:26.904 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:38:26.902+0000 7f244d80aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:26.906 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:38:26.903+0000 7f244d80aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:26.907 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:38:26.905+0000 7f244d80aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:27.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:38:27.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:27.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:38:27.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:27.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:27.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:27.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:27.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:27.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:38:27.225 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:38:27.422 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:38:27.419+0000 7f244d80aec0 -1 Falling back to public interface 2020-03-11T22:38:27.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:38:28.326 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:38:28.323+0000 7f244d80aec0 -1 osd.0 28 log_to_monitors {default=true} 2020-03-11T22:38:28.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:38:28.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:28.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:38:28.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:28.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:38:28.572 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:38:28.922 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/30339,v1:127.0.0.1:6803/30339] [v2:127.0.0.1:6804/30339,v1:127.0.0.1:6805/30339] exists,up 9c7f1786-6f01-4f0d-8a2e-5034ec430e9d 2020-03-11T22:38:28.922 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:38:28.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:28.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:28.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:28.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:516: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:28.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:518: TEST_divergent_2: activate_osd td/divergent-priors 2 2020-03-11T22:38:28.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:38:28.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:38:28.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:38:28.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:38:28.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:38:28.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:38:28.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:38:28.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:38:28.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:38:28.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:38:28.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:38:28.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:38:28.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:38:28.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:38:28.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:38:28.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:38:28.928 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:38:28.928 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:38:28.928 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:38:28.928 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:38:28.929 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:38:28.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:38:28.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:38:28.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:38:28.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:38:28.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:38:28.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:38:28.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:28.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:38:28.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:38:28.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:38:28.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:38:28.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:38:28.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:38:28.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:38:28.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:38:28.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:38:28.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:38:28.944 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:38:28.942+0000 7f1d2afe7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:28.957 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:38:28.955+0000 7f1d2afe7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:28.959 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:38:28.956+0000 7f1d2afe7ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:38:29.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:38:29.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:38:29.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:38:29.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:38:29.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:38:29.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:29.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:38:29.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:29.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:29.267 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:38:29.483 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:38:29.480+0000 7f1d2afe7ec0 -1 Falling back to public interface 2020-03-11T22:38:29.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:38:30.393 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:38:30.391+0000 7f1d2afe7ec0 -1 osd.2 28 log_to_monitors {default=true} 2020-03-11T22:38:30.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:38:30.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:38:30.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:38:30.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:38:30.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:38:30.638 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:38:30.978 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [14,28) [v2:127.0.0.1:6810/30533,v1:127.0.0.1:6811/30533] [v2:127.0.0.1:6812/30533,v1:127.0.0.1:6813/30533] exists,up 9b1fa1a6-0529-400f-aeb3-4b08986f4d72 2020-03-11T22:38:30.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:38:30.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:38:30.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:38:30.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:30.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.0 2020-03-11T22:38:31.541 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already in. 2020-03-11T22:38:31.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:520: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:38:31.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:522: TEST_divergent_2: ceph osd in osd.2 2020-03-11T22:38:32.340 INFO:tasks.workunit.client.0.smithi155.stderr:osd.2 is already in. 2020-03-11T22:38:32.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:38:32.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:525: TEST_divergent_2: wait_for_clean 2020-03-11T22:38:32.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:38:32.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:38:32.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:38:32.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:38:32.349 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:38:32.350 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:38:32.350 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:38:32.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:38:32.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:38:32.350 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:38:32.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:38:32.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:38:32.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:38:32.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:38:32.405 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:38:32.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:38:32.751 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:38:32.751 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:38:32.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:38:32.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:32.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:38:32.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:38:32.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067' 2020-03-11T22:38:32.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:32.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:38:32.993 INFO:tasks.workunit.client.0.smithi155.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:38:33.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:38:33.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1-' 2020-03-11T22:38:33.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:38:33.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:38:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:38:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888067 1- 2-150323855363' 2020-03-11T22:38:33.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:33.131 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888067 2020-03-11T22:38:33.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:33.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:38:33.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888067 2020-03-11T22:38:33.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:33.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:38:33.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888067' 2020-03-11T22:38:33.134 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 146028888067 2020-03-11T22:38:33.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:33.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 146028888067 2020-03-11T22:38:33.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:38:34.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:38:34.481 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:38:34.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888068 -lt 146028888067 2020-03-11T22:38:34.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:34.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:38:34.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:34.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:38:34.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:38:34.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:34.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:38:34.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:38:34.847 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:38:34.847 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 2020-03-11T22:38:35.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705679 -lt 2020-03-11T22:38:35.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705679: unary operator expected 2020-03-11T22:38:35.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:38:35.196 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855363 2020-03-11T22:38:35.196 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:38:35.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:38:35.197 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855363 2020-03-11T22:38:35.197 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:38:35.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:38:35.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855363' 2020-03-11T22:38:35.198 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:38:35.198 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 150323855363 2020-03-11T22:38:35.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855363 -lt 150323855363 2020-03-11T22:38:35.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:38:35.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:35.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:35.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:38:36.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:36.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:36.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:36.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:36.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:36.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:36.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:36.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:36.347 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:36.347 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:36.347 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:36.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:36.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:38:36.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:38:36.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:38:36.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:36.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:38:36.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:36.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:36.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:36.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:36.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:36.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:36.893 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:36.893 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:37.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:37.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:37.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:37.225 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:37.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:37.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:37.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:37.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:37.668 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:37.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:37.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:37.669 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:37.669 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:38.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:38.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:38.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:38:38.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:38.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:38:38.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:38.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:38.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:38.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:38.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:38.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:38.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:38.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:38.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:38.644 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:38.644 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:38.644 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:39.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:39.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:39.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:39.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:39.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:39.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:39.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:39.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:39.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:39.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:39.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:39.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:38:39.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:39.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:38:39.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:39.931 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:39.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:39.931 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:39.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:39.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:39.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:39.932 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:40.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:40.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:40.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:40.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:40.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:40.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:40.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:40.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:40.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:40.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:40.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:40.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:40.707 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:41.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:41.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:41.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:38:41.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:41.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:38:41.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:41.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:41.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:41.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:41.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:41.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:41.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:41.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:42.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:42.309 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:42.309 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:42.309 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:42.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:42.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:42.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:42.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:42.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:42.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:42.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:42.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:42.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:43.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:43.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:43.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:38:43.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:43.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:38:44.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:44.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:44.804 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:44.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:44.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:44.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:44.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:44.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:45.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:45.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:45.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:45.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:45.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:45.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:45.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:45.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:45.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:45.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:45.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:45.590 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:45.590 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:46.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:46.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:46.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:38:46.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:46.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:38:49.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:49.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:49.233 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:49.233 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:49.233 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:49.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:49.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:49.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:49.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:49.577 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:49.577 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:49.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:50.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:50.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:50.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:50.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:50.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:50.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:50.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:50.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:50.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:50.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:50.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:50.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:38:50.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:50.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:38:56.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:38:56.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:38:56.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:38:56.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:38:56.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:38:56.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:38:56.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:38:56.876 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:38:57.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:38:57.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:38:57.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:38:57.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:38:57.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:38:57.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:38:57.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:38:57.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:38:57.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:38:57.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:38:57.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:38:57.652 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:38:57.652 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:38:58.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:38:58.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:38:58.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:38:58.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:38:58.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:39:05.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:05.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:05.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:05.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:05.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:05.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:05.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:05.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:05.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:05.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:05.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:05.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:06.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:06.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:06.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:06.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:06.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:06.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:06.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:06.248 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:06.249 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:06.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:06.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:06.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:39:06.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:39:07.129 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:39:07.129 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_fingerprint": "6c4d9be0-8603-41ec-aaa4-81bc10927acc", 2020-03-11T22:39:07.129 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:39:07.129 INFO:tasks.workunit.client.0.smithi155.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:39:07.130 INFO:tasks.workunit.client.0.smithi155.stdout: "timestamp": "2020-03-11T22:39:07.125360+0000", 2020-03-11T22:39:07.130 INFO:tasks.workunit.client.0.smithi155.stdout: "tag": "", 2020-03-11T22:39:07.130 INFO:tasks.workunit.client.0.smithi155.stdout: "health": { 2020-03-11T22:39:07.130 INFO:tasks.workunit.client.0.smithi155.stdout: "status": "HEALTH_WARN", 2020-03-11T22:39:07.131 INFO:tasks.workunit.client.0.smithi155.stdout: "checks": { 2020-03-11T22:39:07.131 INFO:tasks.workunit.client.0.smithi155.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:39:07.131 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:07.131 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:39:07.132 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:39:07.132 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 17 2020-03-11T22:39:07.132 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.132 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [], 2020-03-11T22:39:07.132 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:39:07.133 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.133 INFO:tasks.workunit.client.0.smithi155.stdout: "OSD_DOWN": { 2020-03-11T22:39:07.133 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:07.133 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:39:07.134 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 osds down", 2020-03-11T22:39:07.134 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:39:07.134 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.134 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:39:07.135 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.135 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "osd.1 (root=default,host=smithi155) is down" 2020-03-11T22:39:07.135 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.135 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.135 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:39:07.136 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.136 INFO:tasks.workunit.client.0.smithi155.stdout: "PG_DEGRADED": { 2020-03-11T22:39:07.136 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:07.136 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:39:07.137 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:39:07.137 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:39:07.137 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.137 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:39:07.138 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.138 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:39:07.138 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.138 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.139 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:39:07.139 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.139 INFO:tasks.workunit.client.0.smithi155.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:39:07.139 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:07.139 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:39:07.140 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:39:07.140 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:39:07.140 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.140 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:39:07.140 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.140 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:39:07.140 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.141 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.141 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:39:07.141 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.141 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.141 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:39:07.141 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.142 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.142 INFO:tasks.workunit.client.0.smithi155.stdout: "mutes": [] 2020-03-11T22:39:07.142 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.142 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_first_committed": 1, 2020-03-11T22:39:07.142 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_last_committed": 1, 2020-03-11T22:39:07.143 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap": { 2020-03-11T22:39:07.143 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:39:07.143 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:39:07.143 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:37:27.148700Z", 2020-03-11T22:39:07.143 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:37:27.148700Z", 2020-03-11T22:39:07.143 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release": 15, 2020-03-11T22:39:07.143 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:39:07.144 INFO:tasks.workunit.client.0.smithi155.stdout: "features": { 2020-03-11T22:39:07.144 INFO:tasks.workunit.client.0.smithi155.stdout: "persistent": [ 2020-03-11T22:39:07.144 INFO:tasks.workunit.client.0.smithi155.stdout: "kraken", 2020-03-11T22:39:07.144 INFO:tasks.workunit.client.0.smithi155.stdout: "luminous", 2020-03-11T22:39:07.144 INFO:tasks.workunit.client.0.smithi155.stdout: "mimic", 2020-03-11T22:39:07.144 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap-prune", 2020-03-11T22:39:07.144 INFO:tasks.workunit.client.0.smithi155.stdout: "nautilus", 2020-03-11T22:39:07.145 INFO:tasks.workunit.client.0.smithi155.stdout: "octopus" 2020-03-11T22:39:07.145 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.145 INFO:tasks.workunit.client.0.smithi155.stdout: "optional": [] 2020-03-11T22:39:07.145 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.145 INFO:tasks.workunit.client.0.smithi155.stdout: "mons": [ 2020-03-11T22:39:07.145 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.145 INFO:tasks.workunit.client.0.smithi155.stdout: "rank": 0, 2020-03-11T22:39:07.146 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "a", 2020-03-11T22:39:07.147 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:39:07.147 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.147 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.147 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.148 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:39:07.148 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 0 2020-03-11T22:39:07.148 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.149 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.149 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.149 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:39:07.149 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:39:07.149 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 0, 2020-03-11T22:39:07.149 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 0 2020-03-11T22:39:07.150 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.150 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.150 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.150 INFO:tasks.workunit.client.0.smithi155.stdout: "quorum": [ 2020-03-11T22:39:07.150 INFO:tasks.workunit.client.0.smithi155.stdout: 0 2020-03-11T22:39:07.150 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.150 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap": { 2020-03-11T22:39:07.151 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 39, 2020-03-11T22:39:07.151 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:39:07.151 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:37:27.895313+0000", 2020-03-11T22:39:07.151 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:38:32.227918+0000", 2020-03-11T22:39:07.151 INFO:tasks.workunit.client.0.smithi155.stdout: "last_up_change": "2020-03-11T22:38:30.474470+0000", 2020-03-11T22:39:07.151 INFO:tasks.workunit.client.0.smithi155.stdout: "last_in_change": "2020-03-11T22:38:32.053975+0000", 2020-03-11T22:39:07.151 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:39:07.152 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_num": 5800384, 2020-03-11T22:39:07.152 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_set": [ 2020-03-11T22:39:07.152 INFO:tasks.workunit.client.0.smithi155.stdout: "nodown", 2020-03-11T22:39:07.152 INFO:tasks.workunit.client.0.smithi155.stdout: "noin", 2020-03-11T22:39:07.152 INFO:tasks.workunit.client.0.smithi155.stdout: "noout", 2020-03-11T22:39:07.152 INFO:tasks.workunit.client.0.smithi155.stdout: "pglog_hardlimit", 2020-03-11T22:39:07.153 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snapdirs", 2020-03-11T22:39:07.153 INFO:tasks.workunit.client.0.smithi155.stdout: "recovery_deletes", 2020-03-11T22:39:07.153 INFO:tasks.workunit.client.0.smithi155.stdout: "sortbitwise" 2020-03-11T22:39:07.153 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.153 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_version": 7, 2020-03-11T22:39:07.153 INFO:tasks.workunit.client.0.smithi155.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:39:07.153 INFO:tasks.workunit.client.0.smithi155.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:39:07.154 INFO:tasks.workunit.client.0.smithi155.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:39:07.154 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_snapshot": "", 2020-03-11T22:39:07.154 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_max": 1, 2020-03-11T22:39:07.154 INFO:tasks.workunit.client.0.smithi155.stdout: "max_osd": 3, 2020-03-11T22:39:07.154 INFO:tasks.workunit.client.0.smithi155.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:39:07.154 INFO:tasks.workunit.client.0.smithi155.stdout: "min_compat_client": "jewel", 2020-03-11T22:39:07.154 INFO:tasks.workunit.client.0.smithi155.stdout: "require_osd_release": "octopus", 2020-03-11T22:39:07.155 INFO:tasks.workunit.client.0.smithi155.stdout: "pools": [ 2020-03-11T22:39:07.155 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.155 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:39:07.155 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_name": "test", 2020-03-11T22:39:07.155 INFO:tasks.workunit.client.0.smithi155.stdout: "create_time": "2020-03-11T22:37:50.690400+0000", 2020-03-11T22:39:07.155 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": 1, 2020-03-11T22:39:07.155 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_names": "hashpspool", 2020-03-11T22:39:07.155 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2020-03-11T22:39:07.156 INFO:tasks.workunit.client.0.smithi155.stdout: "size": 3, 2020-03-11T22:39:07.156 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 2, 2020-03-11T22:39:07.156 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_rule": 0, 2020-03-11T22:39:07.156 INFO:tasks.workunit.client.0.smithi155.stdout: "object_hash": 2, 2020-03-11T22:39:07.156 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:39:07.156 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num": 1, 2020-03-11T22:39:07.156 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num": 1, 2020-03-11T22:39:07.157 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num_target": 1, 2020-03-11T22:39:07.157 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_target": 1, 2020-03-11T22:39:07.157 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_pending": 1, 2020-03-11T22:39:07.157 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pg_merge_meta": { 2020-03-11T22:39:07.157 INFO:tasks.workunit.client.0.smithi155.stdout: "source_pgid": "0.0", 2020-03-11T22:39:07.157 INFO:tasks.workunit.client.0.smithi155.stdout: "ready_epoch": 0, 2020-03-11T22:39:07.158 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_started": 0, 2020-03-11T22:39:07.158 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_clean": 0, 2020-03-11T22:39:07.158 INFO:tasks.workunit.client.0.smithi155.stdout: "source_version": "0'0", 2020-03-11T22:39:07.158 INFO:tasks.workunit.client.0.smithi155.stdout: "target_version": "0'0" 2020-03-11T22:39:07.158 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.158 INFO:tasks.workunit.client.0.smithi155.stdout: "last_change": "28", 2020-03-11T22:39:07.158 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend": "0", 2020-03-11T22:39:07.159 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:39:07.159 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:39:07.159 INFO:tasks.workunit.client.0.smithi155.stdout: "auid": 0, 2020-03-11T22:39:07.159 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:39:07.159 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_seq": 0, 2020-03-11T22:39:07.159 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_epoch": 0, 2020-03-11T22:39:07.160 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_snaps": [], 2020-03-11T22:39:07.161 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps": "[]", 2020-03-11T22:39:07.161 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_bytes": 0, 2020-03-11T22:39:07.161 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_objects": 0, 2020-03-11T22:39:07.161 INFO:tasks.workunit.client.0.smithi155.stdout: "tiers": [], 2020-03-11T22:39:07.161 INFO:tasks.workunit.client.0.smithi155.stdout: "tier_of": -1, 2020-03-11T22:39:07.161 INFO:tasks.workunit.client.0.smithi155.stdout: "read_tier": -1, 2020-03-11T22:39:07.161 INFO:tasks.workunit.client.0.smithi155.stdout: "write_tier": -1, 2020-03-11T22:39:07.162 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_mode": "none", 2020-03-11T22:39:07.162 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_bytes": 0, 2020-03-11T22:39:07.162 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_objects": 0, 2020-03-11T22:39:07.162 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:39:07.162 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:39:07.162 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:39:07.163 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_flush_age": 0, 2020-03-11T22:39:07.163 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_evict_age": 0, 2020-03-11T22:39:07.163 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profile": "", 2020-03-11T22:39:07.163 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_params": { 2020-03-11T22:39:07.163 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "none" 2020-03-11T22:39:07.163 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.163 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_period": 0, 2020-03-11T22:39:07.163 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_count": 0, 2020-03-11T22:39:07.164 INFO:tasks.workunit.client.0.smithi155.stdout: "use_gmt_hitset": true, 2020-03-11T22:39:07.164 INFO:tasks.workunit.client.0.smithi155.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:39:07.164 INFO:tasks.workunit.client.0.smithi155.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:39:07.164 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:39:07.164 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:39:07.164 INFO:tasks.workunit.client.0.smithi155.stdout: "grade_table": [], 2020-03-11T22:39:07.164 INFO:tasks.workunit.client.0.smithi155.stdout: "stripe_width": 0, 2020-03-11T22:39:07.165 INFO:tasks.workunit.client.0.smithi155.stdout: "expected_num_objects": 0, 2020-03-11T22:39:07.165 INFO:tasks.workunit.client.0.smithi155.stdout: "fast_read": false, 2020-03-11T22:39:07.165 INFO:tasks.workunit.client.0.smithi155.stdout: "options": {}, 2020-03-11T22:39:07.165 INFO:tasks.workunit.client.0.smithi155.stdout: "application_metadata": {} 2020-03-11T22:39:07.165 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.165 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.165 INFO:tasks.workunit.client.0.smithi155.stdout: "osds": [ 2020-03-11T22:39:07.166 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.166 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:39:07.166 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "9c7f1786-6f01-4f0d-8a2e-5034ec430e9d", 2020-03-11T22:39:07.166 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:39:07.166 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:39:07.166 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:39:07.166 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:39:07.166 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 5, 2020-03-11T22:39:07.167 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 28, 2020-03-11T22:39:07.167 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 34, 2020-03-11T22:39:07.167 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 38, 2020-03-11T22:39:07.167 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 29, 2020-03-11T22:39:07.167 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:39:07.167 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:39:07.167 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.168 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.168 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.168 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:39:07.168 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:07.168 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.168 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.168 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:07.168 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:39:07.170 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:07.170 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.170 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.170 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.171 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:39:07.171 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.171 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.171 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.171 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:39:07.171 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:07.172 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.172 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.172 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:07.172 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:39:07.172 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:07.172 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.173 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.173 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.173 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:07.173 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.173 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.173 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.173 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:39:07.173 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:07.174 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.174 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.174 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:07.174 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:39:07.174 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:07.174 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.174 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.175 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.175 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:07.175 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.175 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.175 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.175 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:39:07.175 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:07.175 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.176 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.176 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:07.176 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:39:07.176 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:07.176 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.176 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.176 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.177 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6803/30339", 2020-03-11T22:39:07.177 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6805/30339", 2020-03-11T22:39:07.177 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30339", 2020-03-11T22:39:07.177 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30339", 2020-03-11T22:39:07.177 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:39:07.177 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:39:07.177 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:39:07.178 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.178 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.178 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.178 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:39:07.178 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "1b163036-a7f8-46e5-b80e-ffab673e9327", 2020-03-11T22:39:07.178 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 0, 2020-03-11T22:39:07.178 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:39:07.178 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:39:07.179 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:39:07.179 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 0, 2020-03-11T22:39:07.179 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 0, 2020-03-11T22:39:07.179 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 9, 2020-03-11T22:39:07.179 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 22, 2020-03-11T22:39:07.179 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 30, 2020-03-11T22:39:07.179 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:39:07.180 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:39:07.180 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.180 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.180 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.180 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:39:07.180 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:07.180 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.180 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.181 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:07.181 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:39:07.181 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:07.181 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.181 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.181 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.181 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:39:07.181 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.182 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.182 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.182 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:39:07.182 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:07.182 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.182 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.182 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:07.183 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:39:07.183 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:07.183 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.183 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.183 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.183 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:07.183 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.183 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.184 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.184 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:39:07.184 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:07.184 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.184 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.184 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:07.184 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:39:07.184 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:07.185 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.185 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.185 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.185 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:07.185 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.185 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.185 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.186 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:39:07.186 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:07.186 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.186 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.186 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:07.186 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:39:07.186 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:07.187 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.187 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.187 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.187 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/25908", 2020-03-11T22:39:07.187 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/25908", 2020-03-11T22:39:07.187 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25908", 2020-03-11T22:39:07.188 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25908", 2020-03-11T22:39:07.188 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:39:07.188 INFO:tasks.workunit.client.0.smithi155.stdout: "exists" 2020-03-11T22:39:07.188 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.188 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.188 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.188 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:39:07.189 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "9b1fa1a6-0529-400f-aeb3-4b08986f4d72", 2020-03-11T22:39:07.189 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:39:07.189 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:39:07.189 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:39:07.189 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:39:07.189 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 14, 2020-03-11T22:39:07.190 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 28, 2020-03-11T22:39:07.190 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 35, 2020-03-11T22:39:07.190 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 0, 2020-03-11T22:39:07.190 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 31, 2020-03-11T22:39:07.190 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:39:07.190 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:39:07.190 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.190 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.191 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.191 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:39:07.191 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:07.191 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.191 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.191 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:07.191 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:39:07.192 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:07.192 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.192 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.192 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.192 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:39:07.192 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.192 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.192 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.193 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:39:07.193 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:07.193 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.193 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.193 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:07.193 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:39:07.193 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:07.194 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.194 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.194 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.194 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:07.194 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.194 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.194 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.194 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:39:07.195 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:07.195 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.195 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.195 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:07.195 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:39:07.195 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:07.195 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.195 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.196 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.196 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:07.196 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:07.196 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.196 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:07.196 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:39:07.196 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:07.197 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.197 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.197 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:07.197 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:39:07.197 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:07.197 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.197 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.197 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.198 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/30533", 2020-03-11T22:39:07.198 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/30533", 2020-03-11T22:39:07.198 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30533", 2020-03-11T22:39:07.198 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30533", 2020-03-11T22:39:07.198 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:39:07.198 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:39:07.198 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:39:07.199 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.199 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.199 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.199 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_xinfo": [ 2020-03-11T22:39:07.199 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.199 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:39:07.199 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:38:24.303380+0000", 2020-03-11T22:39:07.199 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:39:07.200 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:39:07.200 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:39:07.200 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:39:07.200 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:34.908582+0000", 2020-03-11T22:39:07.200 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:39:07.200 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.200 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.201 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:39:07.201 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:38:24.800501+0000", 2020-03-11T22:39:07.201 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:39:07.201 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:39:07.201 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:39:07.202 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:39:07.202 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:41.129664+0000", 2020-03-11T22:39:07.202 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:39:07.202 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.202 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.202 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:39:07.202 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:38:25.301470+0000", 2020-03-11T22:39:07.203 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:39:07.203 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:39:07.203 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:39:07.203 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:39:07.203 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:47.262472+0000", 2020-03-11T22:39:07.203 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:39:07.203 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.204 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.204 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap": [], 2020-03-11T22:39:07.204 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap_items": [], 2020-03-11T22:39:07.204 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_temp": [], 2020-03-11T22:39:07.204 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_temp": [], 2020-03-11T22:39:07.204 INFO:tasks.workunit.client.0.smithi155.stdout: "blacklist": {}, 2020-03-11T22:39:07.204 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profiles": { 2020-03-11T22:39:07.204 INFO:tasks.workunit.client.0.smithi155.stdout: "default": { 2020-03-11T22:39:07.205 INFO:tasks.workunit.client.0.smithi155.stdout: "k": "2", 2020-03-11T22:39:07.205 INFO:tasks.workunit.client.0.smithi155.stdout: "m": "2", 2020-03-11T22:39:07.205 INFO:tasks.workunit.client.0.smithi155.stdout: "plugin": "jerasure", 2020-03-11T22:39:07.205 INFO:tasks.workunit.client.0.smithi155.stdout: "technique": "reed_sol_van" 2020-03-11T22:39:07.205 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.205 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.205 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps_queue": [], 2020-03-11T22:39:07.206 INFO:tasks.workunit.client.0.smithi155.stdout: "new_removed_snaps": [], 2020-03-11T22:39:07.206 INFO:tasks.workunit.client.0.smithi155.stdout: "new_purged_snaps": [], 2020-03-11T22:39:07.206 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_node_flags": {}, 2020-03-11T22:39:07.206 INFO:tasks.workunit.client.0.smithi155.stdout: "device_class_flags": {} 2020-03-11T22:39:07.206 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.206 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_metadata": [ 2020-03-11T22:39:07.206 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.206 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:39:07.207 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:39:07.207 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6804/30339,v1:127.0.0.1:6805/30339]", 2020-03-11T22:39:07.207 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:39:07.207 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:39:07.207 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:07.207 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:07.207 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:07.208 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:39:07.209 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6802/30339,v1:127.0.0.1:6803/30339]", 2020-03-11T22:39:07.210 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30339,v1:127.0.0.1:6809/30339]", 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30339,v1:127.0.0.1:6807/30339]", 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:07.211 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:07.212 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:07.212 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:39:07.212 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:07.212 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:07.212 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:39:07.212 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:39:07.212 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:07.213 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:39:07.213 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.213 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.213 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:39:07.213 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:39:07.213 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/25908,v1:127.0.0.1:6813/25908]", 2020-03-11T22:39:07.213 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:39:07.213 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:39:07.214 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:07.214 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:07.214 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:07.214 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:07.214 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:07.214 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:07.214 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:07.214 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:39:07.215 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:07.215 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:07.215 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:07.215 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:07.215 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:39:07.215 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:07.216 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:07.216 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:07.216 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:39:07.216 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:39:07.216 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:07.216 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:39:07.216 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:39:07.217 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:07.217 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:39:07.217 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/25908,v1:127.0.0.1:6811/25908]", 2020-03-11T22:39:07.217 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:39:07.217 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25908,v1:127.0.0.1:6817/25908]", 2020-03-11T22:39:07.217 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25908,v1:127.0.0.1:6815/25908]", 2020-03-11T22:39:07.217 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:39:07.218 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:39:07.218 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:07.218 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:07.218 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:07.218 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:39:07.218 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:07.218 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:07.218 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:39:07.219 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:39:07.219 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:07.219 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:39:07.219 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.219 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.219 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:39:07.219 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:39:07.220 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/30533,v1:127.0.0.1:6813/30533]", 2020-03-11T22:39:07.220 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:39:07.220 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:39:07.220 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:07.220 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:07.220 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:07.220 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:07.220 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:07.221 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:07.221 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:07.221 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:39:07.221 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:07.221 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:07.221 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:07.221 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:39:07.222 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:07.223 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:39:07.223 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:39:07.223 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:07.223 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:39:07.223 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/30533,v1:127.0.0.1:6811/30533]", 2020-03-11T22:39:07.223 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:39:07.223 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30533,v1:127.0.0.1:6817/30533]", 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30533,v1:127.0.0.1:6815/30533]", 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:07.224 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:39:07.225 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:07.225 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:07.225 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:39:07.225 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:39:07.225 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:07.225 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:39:07.225 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.225 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.226 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_first_committed": 1, 2020-03-11T22:39:07.226 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_last_committed": 39, 2020-03-11T22:39:07.226 INFO:tasks.workunit.client.0.smithi155.stdout: "crushmap": { 2020-03-11T22:39:07.226 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": [ 2020-03-11T22:39:07.226 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.226 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:39:07.226 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.0", 2020-03-11T22:39:07.227 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:39:07.227 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.227 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.227 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:39:07.227 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.1", 2020-03-11T22:39:07.227 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:39:07.227 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.227 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.228 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:39:07.228 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.2", 2020-03-11T22:39:07.228 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:39:07.228 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.228 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.228 INFO:tasks.workunit.client.0.smithi155.stdout: "types": [ 2020-03-11T22:39:07.228 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.228 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 0, 2020-03-11T22:39:07.229 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd" 2020-03-11T22:39:07.229 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.229 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.229 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:39:07.229 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "host" 2020-03-11T22:39:07.229 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.229 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.229 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 2, 2020-03-11T22:39:07.230 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "chassis" 2020-03-11T22:39:07.230 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.230 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.230 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 3, 2020-03-11T22:39:07.230 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "rack" 2020-03-11T22:39:07.230 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.230 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.231 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 4, 2020-03-11T22:39:07.231 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "row" 2020-03-11T22:39:07.231 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.231 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.231 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 5, 2020-03-11T22:39:07.231 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pdu" 2020-03-11T22:39:07.231 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.231 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.232 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 6, 2020-03-11T22:39:07.232 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pod" 2020-03-11T22:39:07.232 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.232 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.232 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 7, 2020-03-11T22:39:07.232 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "room" 2020-03-11T22:39:07.232 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.232 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.233 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 8, 2020-03-11T22:39:07.233 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "datacenter" 2020-03-11T22:39:07.233 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.233 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.233 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 9, 2020-03-11T22:39:07.233 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "zone" 2020-03-11T22:39:07.233 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.234 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.234 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 10, 2020-03-11T22:39:07.234 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "region" 2020-03-11T22:39:07.234 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.234 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.234 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:39:07.234 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "root" 2020-03-11T22:39:07.234 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.235 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.235 INFO:tasks.workunit.client.0.smithi155.stdout: "buckets": [ 2020-03-11T22:39:07.235 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.235 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -1, 2020-03-11T22:39:07.235 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default", 2020-03-11T22:39:07.235 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:39:07.235 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:39:07.235 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:39:07.236 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:39:07.236 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:39:07.236 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:39:07.236 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.236 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:39:07.236 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:39:07.236 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:39:07.236 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.237 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.237 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.237 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.237 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -2, 2020-03-11T22:39:07.237 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default~hdd", 2020-03-11T22:39:07.237 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:39:07.237 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:39:07.238 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:39:07.238 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:39:07.238 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:39:07.238 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:39:07.238 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.238 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:39:07.238 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:39:07.238 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:39:07.239 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.239 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.239 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.239 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.239 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:39:07.239 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155", 2020-03-11T22:39:07.239 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:39:07.239 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:39:07.240 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:39:07.240 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:39:07.240 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:39:07.240 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:39:07.240 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.240 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:39:07.240 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:39:07.241 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:39:07.241 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.241 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.241 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:39:07.241 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:39:07.241 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:39:07.241 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.241 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.242 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:39:07.242 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:39:07.242 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:39:07.242 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.242 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.242 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.242 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.242 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:39:07.243 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155~hdd", 2020-03-11T22:39:07.243 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:39:07.243 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:39:07.243 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:39:07.243 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:39:07.243 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:39:07.243 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:39:07.244 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.244 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:39:07.244 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:39:07.244 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:39:07.244 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.244 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.244 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:39:07.244 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:39:07.245 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:39:07.245 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.245 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.245 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:39:07.245 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:39:07.245 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:39:07.245 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.246 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.246 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.246 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.246 INFO:tasks.workunit.client.0.smithi155.stdout: "rules": [ 2020-03-11T22:39:07.246 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.246 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_id": 0, 2020-03-11T22:39:07.246 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_name": "replicated_rule", 2020-03-11T22:39:07.246 INFO:tasks.workunit.client.0.smithi155.stdout: "ruleset": 0, 2020-03-11T22:39:07.247 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2020-03-11T22:39:07.247 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 1, 2020-03-11T22:39:07.247 INFO:tasks.workunit.client.0.smithi155.stdout: "max_size": 10, 2020-03-11T22:39:07.247 INFO:tasks.workunit.client.0.smithi155.stdout: "steps": [ 2020-03-11T22:39:07.247 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.247 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "take", 2020-03-11T22:39:07.247 INFO:tasks.workunit.client.0.smithi155.stdout: "item": -1, 2020-03-11T22:39:07.247 INFO:tasks.workunit.client.0.smithi155.stdout: "item_name": "default" 2020-03-11T22:39:07.248 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.248 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.248 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "choose_firstn", 2020-03-11T22:39:07.248 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 0, 2020-03-11T22:39:07.248 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "osd" 2020-03-11T22:39:07.248 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.248 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.248 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "emit" 2020-03-11T22:39:07.249 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.249 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:07.249 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.249 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.249 INFO:tasks.workunit.client.0.smithi155.stdout: "tunables": { 2020-03-11T22:39:07.249 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_tries": 0, 2020-03-11T22:39:07.249 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:39:07.250 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_total_tries": 50, 2020-03-11T22:39:07.250 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:39:07.250 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:39:07.250 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_stable": 1, 2020-03-11T22:39:07.250 INFO:tasks.workunit.client.0.smithi155.stdout: "straw_calc_version": 1, 2020-03-11T22:39:07.250 INFO:tasks.workunit.client.0.smithi155.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:39:07.250 INFO:tasks.workunit.client.0.smithi155.stdout: "profile": "jewel", 2020-03-11T22:39:07.251 INFO:tasks.workunit.client.0.smithi155.stdout: "optimal_tunables": 1, 2020-03-11T22:39:07.251 INFO:tasks.workunit.client.0.smithi155.stdout: "legacy_tunables": 0, 2020-03-11T22:39:07.251 INFO:tasks.workunit.client.0.smithi155.stdout: "minimum_required_version": "jewel", 2020-03-11T22:39:07.251 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables": 1, 2020-03-11T22:39:07.251 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables2": 1, 2020-03-11T22:39:07.251 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v2_rules": 0, 2020-03-11T22:39:07.252 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables3": 1, 2020-03-11T22:39:07.252 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v3_rules": 0, 2020-03-11T22:39:07.252 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v4_buckets": 1, 2020-03-11T22:39:07.252 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables5": 1, 2020-03-11T22:39:07.252 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v5_rules": 0 2020-03-11T22:39:07.252 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.253 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_args": {} 2020-03-11T22:39:07.253 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.253 INFO:tasks.workunit.client.0.smithi155.stdout: "fsmap": { 2020-03-11T22:39:07.253 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:39:07.253 INFO:tasks.workunit.client.0.smithi155.stdout: "default_fscid": -1, 2020-03-11T22:39:07.253 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": { 2020-03-11T22:39:07.253 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": {}, 2020-03-11T22:39:07.253 INFO:tasks.workunit.client.0.smithi155.stdout: "ro_compat": {}, 2020-03-11T22:39:07.254 INFO:tasks.workunit.client.0.smithi155.stdout: "incompat": { 2020-03-11T22:39:07.254 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_1": "base v0.20", 2020-03-11T22:39:07.254 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:39:07.254 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:39:07.254 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:39:07.254 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:39:07.254 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:39:07.255 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_8": "no anchor table", 2020-03-11T22:39:07.255 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_9": "file layout v2", 2020-03-11T22:39:07.255 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:39:07.255 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.255 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.255 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_flags": { 2020-03-11T22:39:07.255 INFO:tasks.workunit.client.0.smithi155.stdout: "enable_multiple": false, 2020-03-11T22:39:07.255 INFO:tasks.workunit.client.0.smithi155.stdout: "ever_enabled_multiple": false 2020-03-11T22:39:07.256 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.256 INFO:tasks.workunit.client.0.smithi155.stdout: "standbys": [], 2020-03-11T22:39:07.256 INFO:tasks.workunit.client.0.smithi155.stdout: "filesystems": [] 2020-03-11T22:39:07.256 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.256 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:39:07.256 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:39:07.256 INFO:tasks.workunit.client.0.smithi155.stdout: "auth": { 2020-03-11T22:39:07.256 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:39:07.257 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 5, 2020-03-11T22:39:07.257 INFO:tasks.workunit.client.0.smithi155.stdout: "num_secrets": 3 2020-03-11T22:39:07.257 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.257 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 1, 2020-03-11T22:39:07.257 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_active": 1, 2020-03-11T22:39:07.257 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_unknown": 0, 2020-03-11T22:39:07.257 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osd": 3, 2020-03-11T22:39:07.258 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_sum": { 2020-03-11T22:39:07.258 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:39:07.258 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 48300, 2020-03-11T22:39:07.258 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 100, 2020-03-11T22:39:07.258 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2020-03-11T22:39:07.258 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 300, 2020-03-11T22:39:07.258 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:39:07.259 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:39:07.259 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 100, 2020-03-11T22:39:07.259 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:39:07.259 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:39:07.259 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 100, 2020-03-11T22:39:07.259 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:39:07.259 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:39:07.259 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:39:07.260 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 100, 2020-03-11T22:39:07.260 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 100, 2020-03-11T22:39:07.260 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:39:07.260 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:39:07.260 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:39:07.260 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:39:07.260 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:39:07.260 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:39:07.261 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:39:07.261 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:39:07.261 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:39:07.261 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:39:07.261 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:39:07.261 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:39:07.261 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:39:07.262 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:39:07.262 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:39:07.262 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:39:07.262 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:39:07.262 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:39:07.262 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:39:07.262 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:39:07.262 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:39:07.263 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:39:07.263 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:39:07.263 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:39:07.263 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:39:07.263 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.263 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:39:07.263 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:39:07.264 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:39:07.264 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:39:07.264 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 0, 2020-03-11T22:39:07.264 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 0, 2020-03-11T22:39:07.264 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:39:07.264 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:07.264 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:39:07.264 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:39:07.265 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:39:07.265 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.265 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 16, 2020-03-11T22:39:07.265 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 16, 2020-03-11T22:39:07.265 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 2, 2020-03-11T22:39:07.265 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 2, 2020-03-11T22:39:07.265 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 0 2020-03-11T22:39:07.265 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.266 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum": { 2020-03-11T22:39:07.266 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:39:07.266 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:39:07.266 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 3, 2020-03-11T22:39:07.266 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:39:07.266 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:39:07.266 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:39:07.266 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:39:07.267 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3179072, 2020-03-11T22:39:07.267 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 33024, 2020-03-11T22:39:07.267 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 13, 2020-03-11T22:39:07.267 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145714, 2020-03-11T22:39:07.267 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311393728, 2020-03-11T22:39:07.267 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:39:07.267 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:39:07.268 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318867177472, 2020-03-11T22:39:07.268 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:39:07.268 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 33816576, 2020-03-11T22:39:07.268 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 404667, 2020-03-11T22:39:07.268 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:39:07.268 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:07.268 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:39:07.269 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 13332, 2020-03-11T22:39:07.269 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221212140 2020-03-11T22:39:07.269 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.269 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:39:07.269 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:39:07.269 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:39:07.269 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2020-03-11T22:39:07.270 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:39:07.270 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:39:07.270 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:39:07.270 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.270 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:39:07.270 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:39:07.270 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:39:07.271 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:39:07.271 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:39:07.271 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.271 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:39:07.271 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:39:07.271 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.271 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum_by_class": { 2020-03-11T22:39:07.272 INFO:tasks.workunit.client.0.smithi155.stdout: "hdd": { 2020-03-11T22:39:07.272 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:39:07.272 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:39:07.272 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 3, 2020-03-11T22:39:07.272 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:39:07.272 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:39:07.272 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:39:07.272 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:39:07.273 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3179072, 2020-03-11T22:39:07.273 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 33024, 2020-03-11T22:39:07.273 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 13, 2020-03-11T22:39:07.273 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145714, 2020-03-11T22:39:07.273 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311393728, 2020-03-11T22:39:07.273 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:39:07.273 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:39:07.274 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318867177472, 2020-03-11T22:39:07.274 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:39:07.274 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 33816576, 2020-03-11T22:39:07.274 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 404667, 2020-03-11T22:39:07.274 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:39:07.274 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:07.274 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:39:07.274 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 13332, 2020-03-11T22:39:07.275 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221212140 2020-03-11T22:39:07.275 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.275 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:39:07.275 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:39:07.275 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:39:07.275 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2020-03-11T22:39:07.275 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:39:07.276 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:39:07.276 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:39:07.276 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.276 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:39:07.276 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:39:07.276 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:39:07.276 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:39:07.276 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:39:07.277 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.277 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:39:07.277 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:39:07.277 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.277 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.277 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_stats": [ 2020-03-11T22:39:07.277 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.277 INFO:tasks.workunit.client.0.smithi155.stdout: "poolid": 1, 2020-03-11T22:39:07.278 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 1, 2020-03-11T22:39:07.278 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:39:07.278 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 48300, 2020-03-11T22:39:07.278 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 100, 2020-03-11T22:39:07.278 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2020-03-11T22:39:07.278 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 300, 2020-03-11T22:39:07.278 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:39:07.279 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:39:07.279 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 100, 2020-03-11T22:39:07.279 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:39:07.279 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:39:07.280 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 100, 2020-03-11T22:39:07.281 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:39:07.281 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:39:07.281 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:39:07.281 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 100, 2020-03-11T22:39:07.281 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 100, 2020-03-11T22:39:07.281 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:39:07.281 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:39:07.282 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:39:07.282 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:39:07.282 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:39:07.282 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:39:07.282 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:39:07.282 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:39:07.282 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:39:07.283 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:39:07.283 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:39:07.283 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:39:07.283 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:39:07.283 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:39:07.283 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:39:07.283 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:39:07.283 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:39:07.284 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:39:07.284 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:39:07.284 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:39:07.284 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:39:07.284 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:39:07.284 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:39:07.284 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:39:07.284 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:39:07.285 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.285 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:39:07.285 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:39:07.285 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:39:07.285 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:39:07.285 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 19529728, 2020-03-11T22:39:07.285 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 143228, 2020-03-11T22:39:07.286 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:39:07.286 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:07.286 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:39:07.286 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:39:07.286 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:39:07.286 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.286 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 16, 2020-03-11T22:39:07.286 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 16, 2020-03-11T22:39:07.287 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 2, 2020-03-11T22:39:07.287 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 2, 2020-03-11T22:39:07.287 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 3 2020-03-11T22:39:07.287 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.287 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.287 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_stats": [ 2020-03-11T22:39:07.287 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.288 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:39:07.288 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 146028888074 2020-03-11T22:39:07.288 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.288 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.288 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:39:07.288 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 38654705679 2020-03-11T22:39:07.289 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.289 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.289 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:39:07.289 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 150323855370 2020-03-11T22:39:07.289 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.289 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.290 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_state": [ 2020-03-11T22:39:07.290 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.290 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "active+undersized+degraded", 2020-03-11T22:39:07.290 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 1 2020-03-11T22:39:07.290 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.290 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.290 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_osd": [ 2020-03-11T22:39:07.290 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.291 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:39:07.291 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:39:07.291 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 1, 2020-03-11T22:39:07.291 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:07.291 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.291 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.291 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:39:07.292 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:39:07.292 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 0, 2020-03-11T22:39:07.292 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:07.292 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.292 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.292 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:39:07.292 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 1, 2020-03-11T22:39:07.292 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 1, 2020-03-11T22:39:07.293 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:07.293 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.293 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.293 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": [ 2020-03-11T22:39:07.293 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:07.293 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:39:07.293 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": {} 2020-03-11T22:39:07.293 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.294 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:07.294 INFO:tasks.workunit.client.0.smithi155.stdout: "servicemap": { 2020-03-11T22:39:07.294 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:39:07.294 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:37:27.895733+0000", 2020-03-11T22:39:07.294 INFO:tasks.workunit.client.0.smithi155.stdout: "services": {} 2020-03-11T22:39:07.294 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:07.294 INFO:tasks.workunit.client.0.smithi155.stdout: "paxos": { 2020-03-11T22:39:07.294 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:39:07.295 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 310, 2020-03-11T22:39:07.295 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pn": 0, 2020-03-11T22:39:07.295 INFO:tasks.workunit.client.0.smithi155.stdout: "accepted_pn": 0 2020-03-11T22:39:07.295 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:07.295 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:39:07.296 INFO:tasks.workunit.client.0.smithi155.stderr:report 1992709919 2020-03-11T22:39:07.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:39:07.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: expr 2 + 2 2020-03-11T22:39:07.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:528: TEST_divergent_2: objname=existing_4 2020-03-11T22:39:07.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:529: TEST_divergent_2: echo 'writing non-divergent object existing_4' 2020-03-11T22:39:07.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:530: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:07.297 INFO:tasks.workunit.client.0.smithi155.stdout:writing non-divergent object existing_4 2020-03-11T22:39:07.461 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:07.461 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:38:32.448591+0000 28'100 39:122 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:39:07.461 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:39:07.462 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:07.462 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:39:07.471 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:531: TEST_divergent_2: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:39:07.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:39:07.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:533: TEST_divergent_2: wait_for_clean 2020-03-11T22:39:07.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:39:07.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:39:07.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:39:07.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:39:07.514 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:39:07.515 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:39:07.515 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:39:07.515 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:39:07.515 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:39:07.516 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:39:07.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:39:07.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:39:07.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:39:07.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:39:07.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:39:07.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:39:07.915 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:39:07.915 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:39:07.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:39:07.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:07.915 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:39:08.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888076 2020-03-11T22:39:08.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888076' 2020-03-11T22:39:08.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:08.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:39:08.158 INFO:tasks.workunit.client.0.smithi155.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:39:08.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:39:08.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888076 1-' 2020-03-11T22:39:08.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:39:08.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:39:08.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855372 2020-03-11T22:39:08.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888076 1- 2-150323855372' 2020-03-11T22:39:08.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:08.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888076 2020-03-11T22:39:08.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:08.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:39:08.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888076 2020-03-11T22:39:08.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:08.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888076 2020-03-11T22:39:08.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888076' 2020-03-11T22:39:08.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:08.299 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 146028888076 2020-03-11T22:39:08.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888074 -lt 146028888076 2020-03-11T22:39:08.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:09.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:39:09.643 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:09.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888074 -lt 146028888076 2020-03-11T22:39:09.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:39:10.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 299 -eq 0 ']' 2020-03-11T22:39:10.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:39:11.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888077 -lt 146028888076 2020-03-11T22:39:11.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:11.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:39:11.347 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:11.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:39:11.347 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:39:11.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:11.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:39:11.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:39:11.349 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 2020-03-11T22:39:11.349 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:39:11.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705679 -lt 2020-03-11T22:39:11.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705679: unary operator expected 2020-03-11T22:39:11.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:39:11.695 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855372 2020-03-11T22:39:11.695 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:39:11.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:39:11.695 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855372 2020-03-11T22:39:11.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:39:11.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855372 2020-03-11T22:39:11.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855372' 2020-03-11T22:39:11.697 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 150323855372 2020-03-11T22:39:11.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:39:12.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855372 -lt 150323855372 2020-03-11T22:39:12.041 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:39:12.041 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:12.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:12.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:39:12.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:12.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:12.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:12.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:12.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:12.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:12.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:12.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:12.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:12.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:12.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:13.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:13.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:39:13.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:39:13.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:39:13.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:13.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:39:13.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:13.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:13.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:13.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:13.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:13.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:13.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:13.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:13.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:13.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:13.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:13.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:14.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:14.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:14.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:14.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:14.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:14.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:14.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:14.161 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:14.161 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:14.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:14.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:14.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:39:14.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:14.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:39:14.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:14.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:14.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:14.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:14.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:14.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:14.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:14.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:15.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:15.136 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:15.136 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:15.136 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:15.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:15.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:15.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:15.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:15.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:15.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:15.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:15.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:15.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:16.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:16.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:16.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:39:16.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:16.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:39:16.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:16.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:16.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:16.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:16.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:16.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:16.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:16.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:16.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:16.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:16.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:16.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:17.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:17.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:17.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:17.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:17.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:17.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:17.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:17.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:17.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:17.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:17.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:17.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:39:17.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:17.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:39:18.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:18.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:18.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:18.448 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:18.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:18.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:18.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:18.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:18.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:18.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:18.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:18.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:19.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:19.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:19.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:19.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:19.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:19.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:19.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:19.232 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:19.232 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:19.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:19.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:19.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:39:19.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:19.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:39:21.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:21.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:21.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:21.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:21.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:21.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:21.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:21.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:21.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:21.616 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:21.616 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:21.616 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:22.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:22.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:22.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:22.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:22.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:22.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:22.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:22.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:22.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:22.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:22.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:22.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:39:22.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:22.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:39:25.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:25.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:25.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:25.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:25.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:25.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:25.725 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:25.725 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:26.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:26.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:26.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:26.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:26.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:26.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:26.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:26.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:26.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:26.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:26.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:26.512 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:26.512 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:26.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:26.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:26.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:39:26.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:26.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:39:33.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:33.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:33.359 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:33.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:33.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:33.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:33.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:33.360 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:33.701 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:33.701 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:33.702 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:33.702 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:34.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:34.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:34.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:34.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:34.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:34.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:34.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:34.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:34.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:34.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:34.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:34.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:39:34.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:39:34.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:39:41.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:39:41.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:39:41.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:39:41.900 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:39:41.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:39:41.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:39:41.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:39:41.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:39:42.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:39:42.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:39:42.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:39:42.242 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:39:42.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:39:42.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:39:42.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:39:42.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:39:42.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:39:42.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:39:42.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:39:42.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:39:42.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:39:43.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:39:43.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:39:43.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:39:43.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:39:43.566 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:39:43.566 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_fingerprint": "6c4d9be0-8603-41ec-aaa4-81bc10927acc", 2020-03-11T22:39:43.566 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:39:43.566 INFO:tasks.workunit.client.0.smithi155.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:39:43.566 INFO:tasks.workunit.client.0.smithi155.stdout: "timestamp": "2020-03-11T22:39:43.562851+0000", 2020-03-11T22:39:43.567 INFO:tasks.workunit.client.0.smithi155.stdout: "tag": "", 2020-03-11T22:39:43.567 INFO:tasks.workunit.client.0.smithi155.stdout: "health": { 2020-03-11T22:39:43.567 INFO:tasks.workunit.client.0.smithi155.stdout: "status": "HEALTH_WARN", 2020-03-11T22:39:43.567 INFO:tasks.workunit.client.0.smithi155.stdout: "checks": { 2020-03-11T22:39:43.567 INFO:tasks.workunit.client.0.smithi155.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:39:43.567 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:43.567 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:39:43.568 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:39:43.568 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 17 2020-03-11T22:39:43.568 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.568 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [], 2020-03-11T22:39:43.568 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:39:43.568 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.568 INFO:tasks.workunit.client.0.smithi155.stdout: "OSD_DOWN": { 2020-03-11T22:39:43.569 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:43.569 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:39:43.569 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 osds down", 2020-03-11T22:39:43.569 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:39:43.569 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.569 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:39:43.569 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.570 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "osd.1 (root=default,host=smithi155) is down" 2020-03-11T22:39:43.570 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.570 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.570 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:39:43.570 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.570 INFO:tasks.workunit.client.0.smithi155.stdout: "PG_DEGRADED": { 2020-03-11T22:39:43.571 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:43.572 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:39:43.573 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 1 pg degraded, 1 pg undersized", 2020-03-11T22:39:43.573 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 2 2020-03-11T22:39:43.573 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.573 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:39:43.573 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.573 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.0 is stuck undersized for 69s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:39:43.573 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.574 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.574 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:39:43.574 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.574 INFO:tasks.workunit.client.0.smithi155.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:39:43.574 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:39:43.574 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:39:43.574 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:39:43.574 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:39:43.575 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.575 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:39:43.575 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.575 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:39:43.575 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.575 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.575 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:39:43.576 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.576 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.576 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:39:43.576 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.576 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.576 INFO:tasks.workunit.client.0.smithi155.stdout: "mutes": [] 2020-03-11T22:39:43.576 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.577 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_first_committed": 1, 2020-03-11T22:39:43.577 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_last_committed": 1, 2020-03-11T22:39:43.577 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap": { 2020-03-11T22:39:43.577 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:39:43.577 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:39:43.577 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:37:27.148700Z", 2020-03-11T22:39:43.577 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:37:27.148700Z", 2020-03-11T22:39:43.578 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release": 15, 2020-03-11T22:39:43.578 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:39:43.578 INFO:tasks.workunit.client.0.smithi155.stdout: "features": { 2020-03-11T22:39:43.578 INFO:tasks.workunit.client.0.smithi155.stdout: "persistent": [ 2020-03-11T22:39:43.578 INFO:tasks.workunit.client.0.smithi155.stdout: "kraken", 2020-03-11T22:39:43.578 INFO:tasks.workunit.client.0.smithi155.stdout: "luminous", 2020-03-11T22:39:43.578 INFO:tasks.workunit.client.0.smithi155.stdout: "mimic", 2020-03-11T22:39:43.578 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap-prune", 2020-03-11T22:39:43.579 INFO:tasks.workunit.client.0.smithi155.stdout: "nautilus", 2020-03-11T22:39:43.579 INFO:tasks.workunit.client.0.smithi155.stdout: "octopus" 2020-03-11T22:39:43.579 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.579 INFO:tasks.workunit.client.0.smithi155.stdout: "optional": [] 2020-03-11T22:39:43.579 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.579 INFO:tasks.workunit.client.0.smithi155.stdout: "mons": [ 2020-03-11T22:39:43.579 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.580 INFO:tasks.workunit.client.0.smithi155.stdout: "rank": 0, 2020-03-11T22:39:43.580 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "a", 2020-03-11T22:39:43.580 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:39:43.580 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.580 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.580 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.580 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:39:43.580 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 0 2020-03-11T22:39:43.581 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.581 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.581 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.581 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:39:43.581 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:39:43.581 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 0, 2020-03-11T22:39:43.581 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 0 2020-03-11T22:39:43.582 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.582 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.582 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.582 INFO:tasks.workunit.client.0.smithi155.stdout: "quorum": [ 2020-03-11T22:39:43.582 INFO:tasks.workunit.client.0.smithi155.stdout: 0 2020-03-11T22:39:43.582 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.582 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap": { 2020-03-11T22:39:43.582 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 39, 2020-03-11T22:39:43.583 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:39:43.583 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:37:27.895313+0000", 2020-03-11T22:39:43.583 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:38:32.227918+0000", 2020-03-11T22:39:43.583 INFO:tasks.workunit.client.0.smithi155.stdout: "last_up_change": "2020-03-11T22:38:30.474470+0000", 2020-03-11T22:39:43.583 INFO:tasks.workunit.client.0.smithi155.stdout: "last_in_change": "2020-03-11T22:38:32.053975+0000", 2020-03-11T22:39:43.583 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:39:43.583 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_num": 5800384, 2020-03-11T22:39:43.584 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_set": [ 2020-03-11T22:39:43.584 INFO:tasks.workunit.client.0.smithi155.stdout: "nodown", 2020-03-11T22:39:43.584 INFO:tasks.workunit.client.0.smithi155.stdout: "noin", 2020-03-11T22:39:43.584 INFO:tasks.workunit.client.0.smithi155.stdout: "noout", 2020-03-11T22:39:43.584 INFO:tasks.workunit.client.0.smithi155.stdout: "pglog_hardlimit", 2020-03-11T22:39:43.584 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snapdirs", 2020-03-11T22:39:43.584 INFO:tasks.workunit.client.0.smithi155.stdout: "recovery_deletes", 2020-03-11T22:39:43.585 INFO:tasks.workunit.client.0.smithi155.stdout: "sortbitwise" 2020-03-11T22:39:43.585 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.585 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_version": 7, 2020-03-11T22:39:43.585 INFO:tasks.workunit.client.0.smithi155.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:39:43.585 INFO:tasks.workunit.client.0.smithi155.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:39:43.585 INFO:tasks.workunit.client.0.smithi155.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:39:43.585 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_snapshot": "", 2020-03-11T22:39:43.585 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_max": 1, 2020-03-11T22:39:43.586 INFO:tasks.workunit.client.0.smithi155.stdout: "max_osd": 3, 2020-03-11T22:39:43.586 INFO:tasks.workunit.client.0.smithi155.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:39:43.586 INFO:tasks.workunit.client.0.smithi155.stdout: "min_compat_client": "jewel", 2020-03-11T22:39:43.586 INFO:tasks.workunit.client.0.smithi155.stdout: "require_osd_release": "octopus", 2020-03-11T22:39:43.586 INFO:tasks.workunit.client.0.smithi155.stdout: "pools": [ 2020-03-11T22:39:43.586 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.586 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:39:43.587 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_name": "test", 2020-03-11T22:39:43.587 INFO:tasks.workunit.client.0.smithi155.stdout: "create_time": "2020-03-11T22:37:50.690400+0000", 2020-03-11T22:39:43.587 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": 1, 2020-03-11T22:39:43.587 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_names": "hashpspool", 2020-03-11T22:39:43.587 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2020-03-11T22:39:43.587 INFO:tasks.workunit.client.0.smithi155.stdout: "size": 3, 2020-03-11T22:39:43.587 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 2, 2020-03-11T22:39:43.587 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_rule": 0, 2020-03-11T22:39:43.588 INFO:tasks.workunit.client.0.smithi155.stdout: "object_hash": 2, 2020-03-11T22:39:43.588 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:39:43.588 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num": 1, 2020-03-11T22:39:43.588 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num": 1, 2020-03-11T22:39:43.588 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num_target": 1, 2020-03-11T22:39:43.588 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_target": 1, 2020-03-11T22:39:43.589 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_pending": 1, 2020-03-11T22:39:43.589 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pg_merge_meta": { 2020-03-11T22:39:43.589 INFO:tasks.workunit.client.0.smithi155.stdout: "source_pgid": "0.0", 2020-03-11T22:39:43.589 INFO:tasks.workunit.client.0.smithi155.stdout: "ready_epoch": 0, 2020-03-11T22:39:43.590 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_started": 0, 2020-03-11T22:39:43.590 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_clean": 0, 2020-03-11T22:39:43.590 INFO:tasks.workunit.client.0.smithi155.stdout: "source_version": "0'0", 2020-03-11T22:39:43.590 INFO:tasks.workunit.client.0.smithi155.stdout: "target_version": "0'0" 2020-03-11T22:39:43.590 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.590 INFO:tasks.workunit.client.0.smithi155.stdout: "last_change": "28", 2020-03-11T22:39:43.590 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend": "0", 2020-03-11T22:39:43.591 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:39:43.591 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:39:43.591 INFO:tasks.workunit.client.0.smithi155.stdout: "auid": 0, 2020-03-11T22:39:43.591 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:39:43.591 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_seq": 0, 2020-03-11T22:39:43.591 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_epoch": 0, 2020-03-11T22:39:43.592 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_snaps": [], 2020-03-11T22:39:43.592 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps": "[]", 2020-03-11T22:39:43.592 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_bytes": 0, 2020-03-11T22:39:43.592 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_objects": 0, 2020-03-11T22:39:43.593 INFO:tasks.workunit.client.0.smithi155.stdout: "tiers": [], 2020-03-11T22:39:43.593 INFO:tasks.workunit.client.0.smithi155.stdout: "tier_of": -1, 2020-03-11T22:39:43.593 INFO:tasks.workunit.client.0.smithi155.stdout: "read_tier": -1, 2020-03-11T22:39:43.593 INFO:tasks.workunit.client.0.smithi155.stdout: "write_tier": -1, 2020-03-11T22:39:43.593 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_mode": "none", 2020-03-11T22:39:43.593 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_bytes": 0, 2020-03-11T22:39:43.593 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_objects": 0, 2020-03-11T22:39:43.594 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:39:43.594 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:39:43.594 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:39:43.594 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_flush_age": 0, 2020-03-11T22:39:43.594 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_evict_age": 0, 2020-03-11T22:39:43.594 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profile": "", 2020-03-11T22:39:43.594 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_params": { 2020-03-11T22:39:43.594 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "none" 2020-03-11T22:39:43.595 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.595 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_period": 0, 2020-03-11T22:39:43.595 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_count": 0, 2020-03-11T22:39:43.595 INFO:tasks.workunit.client.0.smithi155.stdout: "use_gmt_hitset": true, 2020-03-11T22:39:43.595 INFO:tasks.workunit.client.0.smithi155.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:39:43.595 INFO:tasks.workunit.client.0.smithi155.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:39:43.596 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:39:43.596 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:39:43.596 INFO:tasks.workunit.client.0.smithi155.stdout: "grade_table": [], 2020-03-11T22:39:43.596 INFO:tasks.workunit.client.0.smithi155.stdout: "stripe_width": 0, 2020-03-11T22:39:43.596 INFO:tasks.workunit.client.0.smithi155.stdout: "expected_num_objects": 0, 2020-03-11T22:39:43.596 INFO:tasks.workunit.client.0.smithi155.stdout: "fast_read": false, 2020-03-11T22:39:43.597 INFO:tasks.workunit.client.0.smithi155.stdout: "options": {}, 2020-03-11T22:39:43.597 INFO:tasks.workunit.client.0.smithi155.stdout: "application_metadata": {} 2020-03-11T22:39:43.597 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.598 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.598 INFO:tasks.workunit.client.0.smithi155.stdout: "osds": [ 2020-03-11T22:39:43.599 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.599 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:39:43.599 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "9c7f1786-6f01-4f0d-8a2e-5034ec430e9d", 2020-03-11T22:39:43.599 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:39:43.599 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:39:43.599 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:39:43.599 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:39:43.600 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 5, 2020-03-11T22:39:43.600 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 28, 2020-03-11T22:39:43.600 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 34, 2020-03-11T22:39:43.600 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 38, 2020-03-11T22:39:43.600 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 29, 2020-03-11T22:39:43.600 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:39:43.600 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:39:43.601 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.601 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.601 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.601 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:39:43.601 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:43.601 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.601 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.602 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:43.602 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:39:43.602 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:43.602 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.602 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.602 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.602 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:39:43.603 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.603 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.603 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.603 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:39:43.603 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:43.603 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.603 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.604 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:43.604 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:39:43.604 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:43.604 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.604 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.604 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.604 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:43.605 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.605 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.605 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.605 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:39:43.605 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:43.605 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.605 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.606 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:43.606 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:39:43.606 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:43.606 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.606 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.606 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.606 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:43.607 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.607 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.607 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.607 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:39:43.607 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:43.607 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.607 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.608 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:43.608 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:39:43.608 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30339 2020-03-11T22:39:43.608 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.608 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.608 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.608 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6803/30339", 2020-03-11T22:39:43.609 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6805/30339", 2020-03-11T22:39:43.609 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6809/30339", 2020-03-11T22:39:43.609 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6807/30339", 2020-03-11T22:39:43.609 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:39:43.609 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:39:43.609 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:39:43.609 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.610 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.610 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.611 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:39:43.611 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "1b163036-a7f8-46e5-b80e-ffab673e9327", 2020-03-11T22:39:43.611 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 0, 2020-03-11T22:39:43.611 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:39:43.612 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:39:43.612 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:39:43.612 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 0, 2020-03-11T22:39:43.612 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 0, 2020-03-11T22:39:43.613 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 9, 2020-03-11T22:39:43.613 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 22, 2020-03-11T22:39:43.613 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 30, 2020-03-11T22:39:43.613 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:39:43.613 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:39:43.613 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.614 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.614 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.614 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:39:43.614 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:43.614 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.614 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.615 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:43.615 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:39:43.615 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:43.615 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.615 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.615 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.616 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:39:43.616 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.616 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.616 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.616 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:39:43.616 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:43.617 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.617 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.617 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:43.617 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:39:43.617 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:43.617 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.618 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.618 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.618 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:43.618 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.618 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.618 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.618 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:39:43.619 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:43.619 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.619 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.619 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:43.619 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:39:43.619 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:43.620 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.620 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.620 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.620 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:43.620 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.620 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.621 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.621 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:39:43.621 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:43.621 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.621 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.621 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:43.622 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:39:43.622 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 25908 2020-03-11T22:39:43.622 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.622 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.622 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.622 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/25908", 2020-03-11T22:39:43.623 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/25908", 2020-03-11T22:39:43.623 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/25908", 2020-03-11T22:39:43.623 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/25908", 2020-03-11T22:39:43.623 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:39:43.623 INFO:tasks.workunit.client.0.smithi155.stdout: "exists" 2020-03-11T22:39:43.623 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.623 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.624 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.624 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:39:43.624 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "9b1fa1a6-0529-400f-aeb3-4b08986f4d72", 2020-03-11T22:39:43.624 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:39:43.624 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:39:43.624 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:39:43.625 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:39:43.625 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 14, 2020-03-11T22:39:43.625 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 28, 2020-03-11T22:39:43.625 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 35, 2020-03-11T22:39:43.625 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 0, 2020-03-11T22:39:43.626 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 31, 2020-03-11T22:39:43.626 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:39:43.626 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:39:43.626 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.626 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.626 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.626 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:39:43.627 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:43.627 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.627 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.627 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:43.627 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:39:43.627 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:43.628 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.628 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.628 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.628 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:39:43.628 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.629 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.629 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.629 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:39:43.629 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:43.629 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.629 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.629 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:43.630 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:39:43.630 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:43.630 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.630 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.630 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.631 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:39:43.631 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.631 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.631 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.631 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:39:43.631 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:43.632 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.632 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.632 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:43.632 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:39:43.633 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:43.633 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.633 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.633 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.633 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:39:43.633 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:39:43.633 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.634 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:39:43.634 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:39:43.634 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:43.634 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.634 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.634 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:39:43.634 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:39:43.635 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 30533 2020-03-11T22:39:43.635 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.635 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.635 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.635 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/30533", 2020-03-11T22:39:43.635 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/30533", 2020-03-11T22:39:43.635 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/30533", 2020-03-11T22:39:43.636 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/30533", 2020-03-11T22:39:43.636 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:39:43.636 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:39:43.636 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:39:43.636 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.636 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.637 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.637 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_xinfo": [ 2020-03-11T22:39:43.637 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.637 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:39:43.638 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:38:24.303380+0000", 2020-03-11T22:39:43.638 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:39:43.638 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:39:43.638 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:39:43.638 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:39:43.639 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:34.908582+0000", 2020-03-11T22:39:43.639 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:39:43.639 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.639 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.639 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:39:43.639 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:38:24.800501+0000", 2020-03-11T22:39:43.640 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:39:43.640 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:39:43.640 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:39:43.640 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:39:43.640 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:41.129664+0000", 2020-03-11T22:39:43.640 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:39:43.640 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.640 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.641 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:39:43.641 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:38:25.301470+0000", 2020-03-11T22:39:43.641 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:39:43.641 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:39:43.641 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:39:43.641 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:39:43.642 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:37:47.262472+0000", 2020-03-11T22:39:43.642 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:39:43.642 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.642 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.642 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap": [], 2020-03-11T22:39:43.642 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap_items": [], 2020-03-11T22:39:43.642 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_temp": [], 2020-03-11T22:39:43.642 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_temp": [], 2020-03-11T22:39:43.643 INFO:tasks.workunit.client.0.smithi155.stdout: "blacklist": {}, 2020-03-11T22:39:43.643 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profiles": { 2020-03-11T22:39:43.643 INFO:tasks.workunit.client.0.smithi155.stdout: "default": { 2020-03-11T22:39:43.643 INFO:tasks.workunit.client.0.smithi155.stdout: "k": "2", 2020-03-11T22:39:43.643 INFO:tasks.workunit.client.0.smithi155.stdout: "m": "2", 2020-03-11T22:39:43.643 INFO:tasks.workunit.client.0.smithi155.stdout: "plugin": "jerasure", 2020-03-11T22:39:43.643 INFO:tasks.workunit.client.0.smithi155.stdout: "technique": "reed_sol_van" 2020-03-11T22:39:43.644 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.644 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.644 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps_queue": [], 2020-03-11T22:39:43.644 INFO:tasks.workunit.client.0.smithi155.stdout: "new_removed_snaps": [], 2020-03-11T22:39:43.644 INFO:tasks.workunit.client.0.smithi155.stdout: "new_purged_snaps": [], 2020-03-11T22:39:43.644 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_node_flags": {}, 2020-03-11T22:39:43.644 INFO:tasks.workunit.client.0.smithi155.stdout: "device_class_flags": {} 2020-03-11T22:39:43.644 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.645 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_metadata": [ 2020-03-11T22:39:43.645 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.645 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:39:43.645 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:39:43.645 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6804/30339,v1:127.0.0.1:6805/30339]", 2020-03-11T22:39:43.645 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:39:43.645 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:39:43.646 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:43.646 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:43.646 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:43.646 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:43.646 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:43.646 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:43.646 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:43.646 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:39:43.647 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:43.647 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:43.647 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:43.647 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:43.647 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:39:43.647 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:43.647 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:43.648 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:43.648 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:39:43.648 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:39:43.648 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:43.648 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:39:43.648 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:39:43.648 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:43.648 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:39:43.649 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6802/30339,v1:127.0.0.1:6803/30339]", 2020-03-11T22:39:43.649 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:39:43.649 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/30339,v1:127.0.0.1:6809/30339]", 2020-03-11T22:39:43.649 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/30339,v1:127.0.0.1:6807/30339]", 2020-03-11T22:39:43.649 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:39:43.649 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:39:43.650 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:43.650 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:43.650 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:43.650 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:39:43.650 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:43.650 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:43.651 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:39:43.651 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:39:43.651 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:43.651 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:39:43.651 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.651 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.651 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:39:43.651 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:39:43.652 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/25908,v1:127.0.0.1:6813/25908]", 2020-03-11T22:39:43.652 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:39:43.652 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:39:43.652 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:43.652 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:43.652 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:43.652 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:43.652 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:43.653 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:43.653 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:43.653 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:39:43.653 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:43.653 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:43.653 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:43.653 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:43.654 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:39:43.654 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:43.654 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:43.654 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:43.654 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:39:43.654 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:39:43.654 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:43.654 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:39:43.655 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:39:43.655 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:43.655 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:39:43.655 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/25908,v1:127.0.0.1:6811/25908]", 2020-03-11T22:39:43.655 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:39:43.655 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/25908,v1:127.0.0.1:6817/25908]", 2020-03-11T22:39:43.655 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/25908,v1:127.0.0.1:6815/25908]", 2020-03-11T22:39:43.656 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:39:43.656 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:39:43.656 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:43.656 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:43.656 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:43.656 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:39:43.656 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:43.656 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:43.657 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:39:43.657 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:39:43.657 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:43.657 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:39:43.657 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.657 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.657 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:39:43.657 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:39:43.658 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/30533,v1:127.0.0.1:6813/30533]", 2020-03-11T22:39:43.658 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:39:43.658 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:39:43.658 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:39:43.658 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:39:43.658 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:39:43.658 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:39:43.659 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:39:43.659 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:39:43.659 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:39:43.659 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:39:43.659 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:39:43.659 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:39:43.659 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:39:43.659 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:39:43.660 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:39:43.660 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:39:43.660 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:39:43.660 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:39:43.660 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:39:43.660 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:39:43.660 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:39:43.661 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:39:43.661 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:39:43.661 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:39:43.661 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:39:43.661 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/30533,v1:127.0.0.1:6811/30533]", 2020-03-11T22:39:43.661 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:39:43.661 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/30533,v1:127.0.0.1:6817/30533]", 2020-03-11T22:39:43.661 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/30533,v1:127.0.0.1:6815/30533]", 2020-03-11T22:39:43.662 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:39:43.662 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:39:43.662 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:39:43.662 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:39:43.662 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:39:43.662 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:39:43.662 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:39:43.662 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:39:43.663 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:39:43.663 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:39:43.663 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:39:43.663 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:39:43.663 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.663 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.663 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_first_committed": 1, 2020-03-11T22:39:43.663 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_last_committed": 39, 2020-03-11T22:39:43.664 INFO:tasks.workunit.client.0.smithi155.stdout: "crushmap": { 2020-03-11T22:39:43.664 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": [ 2020-03-11T22:39:43.664 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.664 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:39:43.664 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.0", 2020-03-11T22:39:43.664 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:39:43.664 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.665 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.665 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:39:43.665 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.1", 2020-03-11T22:39:43.665 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:39:43.665 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.665 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.665 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:39:43.665 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.2", 2020-03-11T22:39:43.666 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:39:43.666 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.666 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.666 INFO:tasks.workunit.client.0.smithi155.stdout: "types": [ 2020-03-11T22:39:43.666 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.666 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 0, 2020-03-11T22:39:43.666 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd" 2020-03-11T22:39:43.666 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.667 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.667 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:39:43.667 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "host" 2020-03-11T22:39:43.667 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.667 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.667 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 2, 2020-03-11T22:39:43.667 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "chassis" 2020-03-11T22:39:43.668 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.668 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.668 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 3, 2020-03-11T22:39:43.668 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "rack" 2020-03-11T22:39:43.668 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.668 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.668 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 4, 2020-03-11T22:39:43.669 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "row" 2020-03-11T22:39:43.669 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.669 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.669 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 5, 2020-03-11T22:39:43.669 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pdu" 2020-03-11T22:39:43.669 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.669 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.670 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 6, 2020-03-11T22:39:43.670 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pod" 2020-03-11T22:39:43.670 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.670 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.670 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 7, 2020-03-11T22:39:43.670 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "room" 2020-03-11T22:39:43.670 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.671 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.671 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 8, 2020-03-11T22:39:43.671 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "datacenter" 2020-03-11T22:39:43.671 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.671 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.671 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 9, 2020-03-11T22:39:43.671 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "zone" 2020-03-11T22:39:43.671 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.672 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.672 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 10, 2020-03-11T22:39:43.672 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "region" 2020-03-11T22:39:43.672 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.672 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.672 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:39:43.672 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "root" 2020-03-11T22:39:43.672 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.673 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.673 INFO:tasks.workunit.client.0.smithi155.stdout: "buckets": [ 2020-03-11T22:39:43.673 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.673 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -1, 2020-03-11T22:39:43.673 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default", 2020-03-11T22:39:43.673 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:39:43.673 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:39:43.674 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:39:43.674 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:39:43.674 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:39:43.674 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:39:43.674 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.674 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:39:43.674 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:39:43.674 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:39:43.675 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.675 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.675 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.675 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.675 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -2, 2020-03-11T22:39:43.675 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default~hdd", 2020-03-11T22:39:43.675 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:39:43.676 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:39:43.676 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:39:43.676 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:39:43.676 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:39:43.676 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:39:43.676 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.676 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:39:43.677 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:39:43.677 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:39:43.677 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.677 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.677 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.677 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.677 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:39:43.677 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155", 2020-03-11T22:39:43.678 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:39:43.678 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:39:43.678 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:39:43.678 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:39:43.678 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:39:43.678 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:39:43.678 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.679 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:39:43.679 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:39:43.679 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:39:43.679 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.679 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.679 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:39:43.679 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:39:43.679 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:39:43.680 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.680 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.680 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:39:43.680 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:39:43.680 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:39:43.680 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.680 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.681 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.681 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.681 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:39:43.681 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155~hdd", 2020-03-11T22:39:43.681 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:39:43.681 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:39:43.681 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:39:43.682 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:39:43.682 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:39:43.682 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:39:43.682 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.682 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:39:43.682 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:39:43.682 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:39:43.682 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.683 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.683 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:39:43.683 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:39:43.683 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:39:43.683 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.683 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.683 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:39:43.683 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:39:43.684 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:39:43.684 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.684 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.684 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.684 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.684 INFO:tasks.workunit.client.0.smithi155.stdout: "rules": [ 2020-03-11T22:39:43.684 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.685 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_id": 0, 2020-03-11T22:39:43.685 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_name": "replicated_rule", 2020-03-11T22:39:43.685 INFO:tasks.workunit.client.0.smithi155.stdout: "ruleset": 0, 2020-03-11T22:39:43.685 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2020-03-11T22:39:43.685 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 1, 2020-03-11T22:39:43.685 INFO:tasks.workunit.client.0.smithi155.stdout: "max_size": 10, 2020-03-11T22:39:43.685 INFO:tasks.workunit.client.0.smithi155.stdout: "steps": [ 2020-03-11T22:39:43.686 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.686 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "take", 2020-03-11T22:39:43.686 INFO:tasks.workunit.client.0.smithi155.stdout: "item": -1, 2020-03-11T22:39:43.686 INFO:tasks.workunit.client.0.smithi155.stdout: "item_name": "default" 2020-03-11T22:39:43.686 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.686 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.686 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "choose_firstn", 2020-03-11T22:39:43.686 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 0, 2020-03-11T22:39:43.687 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "osd" 2020-03-11T22:39:43.687 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.687 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.687 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "emit" 2020-03-11T22:39:43.687 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.687 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:39:43.687 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.687 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.688 INFO:tasks.workunit.client.0.smithi155.stdout: "tunables": { 2020-03-11T22:39:43.688 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_tries": 0, 2020-03-11T22:39:43.688 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:39:43.688 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_total_tries": 50, 2020-03-11T22:39:43.688 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:39:43.688 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:39:43.688 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_stable": 1, 2020-03-11T22:39:43.689 INFO:tasks.workunit.client.0.smithi155.stdout: "straw_calc_version": 1, 2020-03-11T22:39:43.689 INFO:tasks.workunit.client.0.smithi155.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:39:43.689 INFO:tasks.workunit.client.0.smithi155.stdout: "profile": "jewel", 2020-03-11T22:39:43.689 INFO:tasks.workunit.client.0.smithi155.stdout: "optimal_tunables": 1, 2020-03-11T22:39:43.689 INFO:tasks.workunit.client.0.smithi155.stdout: "legacy_tunables": 0, 2020-03-11T22:39:43.689 INFO:tasks.workunit.client.0.smithi155.stdout: "minimum_required_version": "jewel", 2020-03-11T22:39:43.689 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables": 1, 2020-03-11T22:39:43.689 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables2": 1, 2020-03-11T22:39:43.690 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v2_rules": 0, 2020-03-11T22:39:43.690 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables3": 1, 2020-03-11T22:39:43.690 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v3_rules": 0, 2020-03-11T22:39:43.690 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v4_buckets": 1, 2020-03-11T22:39:43.690 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables5": 1, 2020-03-11T22:39:43.690 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v5_rules": 0 2020-03-11T22:39:43.690 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.691 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_args": {} 2020-03-11T22:39:43.691 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.691 INFO:tasks.workunit.client.0.smithi155.stdout: "fsmap": { 2020-03-11T22:39:43.691 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:39:43.691 INFO:tasks.workunit.client.0.smithi155.stdout: "default_fscid": -1, 2020-03-11T22:39:43.691 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": { 2020-03-11T22:39:43.691 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": {}, 2020-03-11T22:39:43.691 INFO:tasks.workunit.client.0.smithi155.stdout: "ro_compat": {}, 2020-03-11T22:39:43.692 INFO:tasks.workunit.client.0.smithi155.stdout: "incompat": { 2020-03-11T22:39:43.692 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_1": "base v0.20", 2020-03-11T22:39:43.692 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:39:43.692 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:39:43.692 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:39:43.692 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:39:43.692 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:39:43.693 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_8": "no anchor table", 2020-03-11T22:39:43.693 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_9": "file layout v2", 2020-03-11T22:39:43.693 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:39:43.693 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.693 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.693 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_flags": { 2020-03-11T22:39:43.693 INFO:tasks.workunit.client.0.smithi155.stdout: "enable_multiple": false, 2020-03-11T22:39:43.693 INFO:tasks.workunit.client.0.smithi155.stdout: "ever_enabled_multiple": false 2020-03-11T22:39:43.694 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.694 INFO:tasks.workunit.client.0.smithi155.stdout: "standbys": [], 2020-03-11T22:39:43.694 INFO:tasks.workunit.client.0.smithi155.stdout: "filesystems": [] 2020-03-11T22:39:43.694 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.694 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:39:43.694 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:39:43.694 INFO:tasks.workunit.client.0.smithi155.stdout: "auth": { 2020-03-11T22:39:43.694 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:39:43.695 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 5, 2020-03-11T22:39:43.695 INFO:tasks.workunit.client.0.smithi155.stdout: "num_secrets": 3 2020-03-11T22:39:43.695 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.695 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 1, 2020-03-11T22:39:43.695 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_active": 1, 2020-03-11T22:39:43.695 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_unknown": 0, 2020-03-11T22:39:43.695 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osd": 3, 2020-03-11T22:39:43.696 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_sum": { 2020-03-11T22:39:43.696 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:39:43.696 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 47947, 2020-03-11T22:39:43.696 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 100, 2020-03-11T22:39:43.696 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2020-03-11T22:39:43.696 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 300, 2020-03-11T22:39:43.696 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:39:43.697 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:39:43.697 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 100, 2020-03-11T22:39:43.697 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:39:43.697 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:39:43.697 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 100, 2020-03-11T22:39:43.697 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:39:43.697 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:39:43.697 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:39:43.698 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 101, 2020-03-11T22:39:43.698 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 101, 2020-03-11T22:39:43.698 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:39:43.698 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:39:43.698 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:39:43.698 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:39:43.698 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:39:43.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:39:43.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:39:43.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:39:43.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:39:43.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:39:43.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:39:43.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:39:43.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:39:43.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:39:43.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:39:43.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:39:43.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:39:43.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:39:43.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:39:43.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:39:43.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:39:43.701 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:39:43.701 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:39:43.701 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:39:43.701 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:39:43.701 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.701 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:39:43.701 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:39:43.702 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:39:43.702 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:39:43.702 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 0, 2020-03-11T22:39:43.702 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 0, 2020-03-11T22:39:43.702 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:39:43.702 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:43.702 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:39:43.702 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:39:43.703 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:39:43.703 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.703 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 17, 2020-03-11T22:39:43.703 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 17, 2020-03-11T22:39:43.703 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 2, 2020-03-11T22:39:43.703 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 2, 2020-03-11T22:39:43.703 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 0 2020-03-11T22:39:43.703 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.704 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum": { 2020-03-11T22:39:43.704 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:39:43.704 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:39:43.704 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 3, 2020-03-11T22:39:43.704 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:39:43.704 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:39:43.704 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:39:43.704 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:39:43.705 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3179072, 2020-03-11T22:39:43.705 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 33024, 2020-03-11T22:39:43.705 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 13, 2020-03-11T22:39:43.705 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145714, 2020-03-11T22:39:43.705 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311393728, 2020-03-11T22:39:43.705 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:39:43.705 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:39:43.706 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318867177472, 2020-03-11T22:39:43.706 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:39:43.706 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 33816576, 2020-03-11T22:39:43.706 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 403961, 2020-03-11T22:39:43.706 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:39:43.706 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:43.706 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:39:43.707 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 13332, 2020-03-11T22:39:43.707 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221212140 2020-03-11T22:39:43.707 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.707 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:39:43.707 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:39:43.707 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:39:43.707 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2020-03-11T22:39:43.707 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:39:43.708 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:39:43.708 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:39:43.708 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.708 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:39:43.708 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:39:43.708 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:39:43.708 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:39:43.709 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:39:43.709 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.709 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:39:43.709 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:39:43.709 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.709 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum_by_class": { 2020-03-11T22:39:43.709 INFO:tasks.workunit.client.0.smithi155.stdout: "hdd": { 2020-03-11T22:39:43.709 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:39:43.710 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:39:43.710 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 3, 2020-03-11T22:39:43.710 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:39:43.710 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:39:43.710 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:39:43.710 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:39:43.710 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3179072, 2020-03-11T22:39:43.711 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 33024, 2020-03-11T22:39:43.711 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 13, 2020-03-11T22:39:43.711 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145714, 2020-03-11T22:39:43.711 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311393728, 2020-03-11T22:39:43.711 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:39:43.711 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:39:43.711 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318867177472, 2020-03-11T22:39:43.711 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:39:43.712 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 33816576, 2020-03-11T22:39:43.712 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 403961, 2020-03-11T22:39:43.712 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:39:43.712 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:43.712 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:39:43.712 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 13332, 2020-03-11T22:39:43.712 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221212140 2020-03-11T22:39:43.713 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.713 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:39:43.713 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:39:43.713 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:39:43.713 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2020-03-11T22:39:43.713 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:39:43.713 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:39:43.714 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:39:43.714 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.714 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:39:43.714 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:39:43.714 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:39:43.714 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:39:43.714 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:39:43.714 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.715 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:39:43.715 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:39:43.715 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.715 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.715 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_stats": [ 2020-03-11T22:39:43.715 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.715 INFO:tasks.workunit.client.0.smithi155.stdout: "poolid": 1, 2020-03-11T22:39:43.715 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 1, 2020-03-11T22:39:43.716 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:39:43.716 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 47947, 2020-03-11T22:39:43.716 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 100, 2020-03-11T22:39:43.716 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2020-03-11T22:39:43.716 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 300, 2020-03-11T22:39:43.716 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:39:43.716 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:39:43.717 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 100, 2020-03-11T22:39:43.717 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:39:43.717 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:39:43.717 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 100, 2020-03-11T22:39:43.717 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:39:43.717 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:39:43.717 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:39:43.717 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 101, 2020-03-11T22:39:43.718 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 101, 2020-03-11T22:39:43.718 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:39:43.718 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:39:43.718 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:39:43.718 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:39:43.718 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:39:43.718 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:39:43.719 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:39:43.719 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:39:43.719 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:39:43.719 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:39:43.719 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:39:43.719 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:39:43.719 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:39:43.719 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:39:43.720 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:39:43.720 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:39:43.720 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:39:43.720 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:39:43.720 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:39:43.720 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:39:43.720 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:39:43.721 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:39:43.721 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:39:43.721 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:39:43.721 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:39:43.721 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.721 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:39:43.721 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:39:43.721 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:39:43.722 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:39:43.722 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 19529728, 2020-03-11T22:39:43.722 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 142522, 2020-03-11T22:39:43.722 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:39:43.722 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:39:43.722 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:39:43.722 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:39:43.722 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:39:43.723 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.723 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 17, 2020-03-11T22:39:43.723 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 17, 2020-03-11T22:39:43.723 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 2, 2020-03-11T22:39:43.723 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 2, 2020-03-11T22:39:43.723 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 3 2020-03-11T22:39:43.723 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.724 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.724 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_stats": [ 2020-03-11T22:39:43.724 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.724 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:39:43.724 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 146028888083 2020-03-11T22:39:43.724 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.724 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.724 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:39:43.725 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 38654705679 2020-03-11T22:39:43.725 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.725 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.725 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:39:43.725 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 150323855379 2020-03-11T22:39:43.725 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.725 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.726 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_state": [ 2020-03-11T22:39:43.726 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.726 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "active+undersized+degraded", 2020-03-11T22:39:43.726 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 1 2020-03-11T22:39:43.726 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.726 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.726 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_osd": [ 2020-03-11T22:39:43.726 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.727 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:39:43.727 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:39:43.727 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 1, 2020-03-11T22:39:43.727 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:43.727 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.727 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.727 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:39:43.728 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:39:43.728 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 0, 2020-03-11T22:39:43.728 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:43.728 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.728 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.728 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:39:43.728 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 1, 2020-03-11T22:39:43.728 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 1, 2020-03-11T22:39:43.729 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:39:43.729 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.729 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.729 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": [ 2020-03-11T22:39:43.729 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:39:43.729 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:39:43.729 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": {} 2020-03-11T22:39:43.730 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.730 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:39:43.730 INFO:tasks.workunit.client.0.smithi155.stdout: "servicemap": { 2020-03-11T22:39:43.730 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:39:43.730 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:37:27.895733+0000", 2020-03-11T22:39:43.730 INFO:tasks.workunit.client.0.smithi155.stdout: "services": {} 2020-03-11T22:39:43.730 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:39:43.730 INFO:tasks.workunit.client.0.smithi155.stdout: "paxos": { 2020-03-11T22:39:43.731 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:39:43.731 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 389, 2020-03-11T22:39:43.731 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pn": 0, 2020-03-11T22:39:43.731 INFO:tasks.workunit.client.0.smithi155.stdout: "accepted_pn": 0 2020-03-11T22:39:43.731 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:39:43.731 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:39:43.732 INFO:tasks.workunit.client.0.smithi155.stderr:report 1871185855 2020-03-11T22:39:43.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:39:43.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:536: TEST_divergent_2: echo 'delay recovery' 2020-03-11T22:39:43.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:537: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:43.733 INFO:tasks.workunit.client.0.smithi155.stdout:delay recovery 2020-03-11T22:39:43.900 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:43.901 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:38:32.448591+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:39:43.901 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:39:43.901 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:43.902 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:39:43.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:39:43.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.0 2020-03-11T22:39:43.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:39:43.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:39:43.912 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:43.912 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:43.912 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:39:43.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.0.asok 2020-03-11T22:39:43.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:39:43.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:39:44.003 INFO:tasks.workunit.client.0.smithi155.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:39:44.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:538: TEST_divergent_2: for i in $non_divergent 2020-03-11T22:39:44.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: get_asok_path osd.2 2020-03-11T22:39:44.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:39:44.011 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:39:44.011 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:44.011 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:44.012 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:39:44.012 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.2.asok 2020-03-11T22:39:44.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:39:44.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:540: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:39:44.104 INFO:tasks.workunit.client.0.smithi155.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:39:44.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:544: TEST_divergent_2: echo 'revive divergent 1' 2020-03-11T22:39:44.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:545: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:44.112 INFO:tasks.workunit.client.0.smithi155.stdout:revive divergent 1 2020-03-11T22:39:44.430 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:44.430 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:38:32.448591+0000 39'101 39:123 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:39:44.430 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:39:44.430 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:44.431 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:39:44.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:546: TEST_divergent_2: ceph osd set noup 2020-03-11T22:39:45.080 INFO:tasks.workunit.client.0.smithi155.stderr:noup is set 2020-03-11T22:39:45.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:547: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:39:45.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:39:45.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:39:45.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:39:45.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:39:45.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:39:45.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:39:45.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:39:45.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:39:45.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:39:45.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:39:45.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:39:45.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:39:45.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:39:45.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:39:45.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:39:45.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:39:45.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:39:45.093 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:39:45.093 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:45.093 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:39:45.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:39:45.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:39:45.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:39:45.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:39:45.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:39:45.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:39:45.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:39:45.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:45.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:39:45.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:39:45.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:39:45.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:39:45.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:39:45.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:39:45.095 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:39:45.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:39:45.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:39:45.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:39:45.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:39:45.117 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:39:45.114+0000 7fc0cc41cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:45.118 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:39:45.116+0000 7fc0cc41cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:45.121 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:39:45.118+0000 7fc0cc41cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:39:45.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:39:45.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:548: TEST_divergent_2: sleep 5 2020-03-11T22:39:45.636 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:39:45.633+0000 7fc0cc41cec0 -1 Falling back to public interface 2020-03-11T22:39:46.501 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:39:46.499+0000 7fc0cc41cec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-11T22:39:50.438 INFO:tasks.workunit.client.0.smithi155.stdout:delay recovery divergent 2020-03-11T22:39:50.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:550: TEST_divergent_2: echo 'delay recovery divergent' 2020-03-11T22:39:50.439 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:551: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:50.767 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:50.768 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:38:32.448591+0000 39'101 39:124 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:39:50.768 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:39:50.768 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:50.769 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:39:50.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:39:50.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:39:50.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:39:50.778 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:39:50.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:39:50.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:39:50.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.1.asok 2020-03-11T22:39:50.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:39:50.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:552: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:39:50.870 INFO:tasks.workunit.client.0.smithi155.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:39:50.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:554: TEST_divergent_2: ceph osd unset noup 2020-03-11T22:39:51.483 INFO:tasks.workunit.client.0.smithi155.stderr:noup is unset 2020-03-11T22:39:51.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:556: TEST_divergent_2: wait_for_osd up 0 2020-03-11T22:39:51.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:51.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:39:51.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:51.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:51.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:51.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:51.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:51.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:39:51.493 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:39:51.874 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 34 up_thru 38 down_at 29 last_clean_interval [5,28) [v2:127.0.0.1:6802/30339,v1:127.0.0.1:6803/30339] [v2:127.0.0.1:6804/30339,v1:127.0.0.1:6805/30339] exists,up 9c7f1786-6f01-4f0d-8a2e-5034ec430e9d 2020-03-11T22:39:51.875 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:39:51.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:51.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:51.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:51.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:557: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:39:51.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:51.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:39:51.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:51.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:51.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:51.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:51.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:51.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:39:52.226 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6818/33107,v1:127.0.0.1:6819/33107] [v2:127.0.0.1:6820/33107,v1:127.0.0.1:6821/33107] exists,up 1b163036-a7f8-46e5-b80e-ffab673e9327 2020-03-11T22:39:52.226 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:39:52.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:52.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:52.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:52.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:558: TEST_divergent_2: wait_for_osd up 2 2020-03-11T22:39:52.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:39:52.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:39:52.227 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:39:52.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:39:52.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:39:52.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:39:52.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:39:52.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:39:52.595 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [14,28) [v2:127.0.0.1:6810/30533,v1:127.0.0.1:6811/30533] [v2:127.0.0.1:6812/30533,v1:127.0.0.1:6813/30533] exists,up 9b1fa1a6-0529-400f-aeb3-4b08986f4d72 2020-03-11T22:39:52.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:39:52.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:39:52.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:39:52.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:560: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:52.973 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:52.973 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:38:32.448591+0000 39'101 39:124 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:39:52.973 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:39:52.974 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:52.975 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:39:52.983 INFO:tasks.workunit.client.0.smithi155.stdout:wait for peering 2020-03-11T22:39:52.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:561: TEST_divergent_2: echo 'wait for peering' 2020-03-11T22:39:52.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:562: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:39:53.303 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:39:53.303 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 47947 0 0 17 17 active+undersized+degraded 2020-03-11T22:38:32.448591+0000 39'101 39:124 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:39:53.304 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:39:53.304 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:39:53.305 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:39:53.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:563: TEST_divergent_2: rados -p test put foo /etc/fstab 2020-03-11T22:40:00.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:567: TEST_divergent_2: echo 'killing divergent 1' 2020-03-11T22:40:00.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:568: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:00.481 INFO:tasks.workunit.client.0.smithi155.stdout:killing divergent 1 2020-03-11T22:40:00.809 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:00.810 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:39:51.806979+0000 39'101 44:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:40:00.810 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:40:00.810 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:00.811 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:40:00.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:569: TEST_divergent_2: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:40:00.820 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:00.820 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:00.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:00.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:00.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:00.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:00.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:572: TEST_divergent_2: expfile=td/divergent-priors/exp.17464.out 2020-03-11T22:40:00.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:573: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17464.out 2020-03-11T22:40:00.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:40:00.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:40:00.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:40:00.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:40:00.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:40:00.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17464.out 2020-03-11T22:40:01.530 INFO:tasks.workunit.client.0.smithi155.stderr:Exporting 1.0 info 1.0( v 44'102 lc 0'0 (28'90,44'102] local-lis/les=43/44 n=101 ec=22/22 lis/c=43/22 les/c/f=44/23/0 sis=43 pruub=8.645268676s) 2020-03-11T22:40:01.538 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:40:01.556 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:40:01.572 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:40:01.582 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:091f9f1d:::existing_94:head# 2020-03-11T22:40:01.583 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:0a865a93:::existing_54:head# 2020-03-11T22:40:01.588 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:0fed513c:::existing_41:head# 2020-03-11T22:40:01.588 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:13b9e0b8:::existing_92:head# 2020-03-11T22:40:01.589 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:169157fa:::existing_81:head# 2020-03-11T22:40:01.590 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:17731382:::existing_58:head# 2020-03-11T22:40:01.590 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:18a45a39:::existing_43:head# 2020-03-11T22:40:01.591 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:1c160f01:::existing_66:head# 2020-03-11T22:40:01.592 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:1dd6c249:::existing_97:head# 2020-03-11T22:40:01.593 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:227e0c77:::existing_7:head# 2020-03-11T22:40:01.593 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:23e1b063:::existing_59:head# 2020-03-11T22:40:01.594 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:28566edb:::existing_78:head# 2020-03-11T22:40:01.594 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:2a101182:::existing_20:head# 2020-03-11T22:40:01.601 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:2d1519da:::existing_96:head# 2020-03-11T22:40:01.602 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:3078b87a:::existing_86:head# 2020-03-11T22:40:01.603 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:30bde01b:::existing_15:head# 2020-03-11T22:40:01.604 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:317a7e19:::existing_56:head# 2020-03-11T22:40:01.605 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:38b9c7fa:::existing_33:head# 2020-03-11T22:40:01.614 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:3e16d84b:::existing_48:head# 2020-03-11T22:40:01.619 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:45fe8203:::existing_75:head# 2020-03-11T22:40:01.624 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:4e82c9ba:::existing_50:head# 2020-03-11T22:40:01.624 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:5298dcdb:::existing_57:head# 2020-03-11T22:40:01.625 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:52ebaa44:::existing_32:head# 2020-03-11T22:40:01.630 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:53588912:::existing_9:head# 2020-03-11T22:40:01.631 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:5b7e2de6:::existing_84:head# 2020-03-11T22:40:01.632 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:5da0255c:::existing_13:head# 2020-03-11T22:40:01.632 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:5da2dab8:::existing_53:head# 2020-03-11T22:40:01.633 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:5ee3514b:::existing_30:head# 2020-03-11T22:40:01.634 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:602f83fe:::foo:head# 2020-03-11T22:40:01.639 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:60426cc7:::existing_62:head# 2020-03-11T22:40:01.640 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:628918ca:::existing_79:head# 2020-03-11T22:40:01.640 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:63f8ab2d:::existing_77:head# 2020-03-11T22:40:01.640 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:67e1780f:::existing_27:head# 2020-03-11T22:40:01.645 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:68143ca1:::existing_71:head# 2020-03-11T22:40:01.648 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:6bac9509:::existing_29:head# 2020-03-11T22:40:01.654 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:6bbec798:::existing_45:head# 2020-03-11T22:40:01.655 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:6cb23a40:::existing_65:head# 2020-03-11T22:40:01.655 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:7055ec5e:::existing_83:head# 2020-03-11T22:40:01.656 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:70a54e2b:::existing_87:head# 2020-03-11T22:40:01.656 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:750ace21:::existing_19:head# 2020-03-11T22:40:01.657 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:78623b1d:::existing_6:head# 2020-03-11T22:40:01.658 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:79881152:::existing_63:head# 2020-03-11T22:40:01.658 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:7a80c0d5:::existing_99:head# 2020-03-11T22:40:01.659 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:81490677:::existing_76:head# 2020-03-11T22:40:01.659 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:85ec8e5d:::existing_85:head# 2020-03-11T22:40:01.660 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:8e2abbf0:::existing_49:head# 2020-03-11T22:40:01.660 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:40:01.660 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:923c5d4d:::existing_5:head# 2020-03-11T22:40:01.661 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:92731e71:::existing_89:head# 2020-03-11T22:40:01.661 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:928205cf:::existing_82:head# 2020-03-11T22:40:01.662 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:93be2267:::existing_24:head# 2020-03-11T22:40:01.662 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:99ceda2c:::existing_44:head# 2020-03-11T22:40:01.663 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:9dd917ee:::existing_26:head# 2020-03-11T22:40:01.663 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:a28805a6:::existing_72:head# 2020-03-11T22:40:01.664 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:a294a8f6:::existing_16:head# 2020-03-11T22:40:01.664 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:a2f5473a:::existing_42:head# 2020-03-11T22:40:01.664 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:a504a533:::existing_90:head# 2020-03-11T22:40:01.665 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:a691337e:::existing_61:head# 2020-03-11T22:40:01.666 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:40:01.666 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:a9fde86e:::existing_18:head# 2020-03-11T22:40:01.666 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:aa309faa:::existing_68:head# 2020-03-11T22:40:01.667 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:ae40b0ed:::existing_31:head# 2020-03-11T22:40:01.667 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:b279b1fe:::existing_10:head# 2020-03-11T22:40:01.667 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:b343e277:::existing_88:head# 2020-03-11T22:40:01.668 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:b7130e90:::existing_80:head# 2020-03-11T22:40:01.668 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:bc6f7309:::existing_70:head# 2020-03-11T22:40:01.672 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:be8489b7:::existing_38:head# 2020-03-11T22:40:01.673 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:beb4e5e3:::existing_47:head# 2020-03-11T22:40:01.673 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:bebec877:::existing_74:head# 2020-03-11T22:40:01.674 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:c04347d8:::existing_60:head# 2020-03-11T22:40:01.674 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:c156033e:::existing_100:head# 2020-03-11T22:40:01.675 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:c19f5a8e:::existing_8:head# 2020-03-11T22:40:01.676 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:40:01.676 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:40:01.677 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:cc1d0abf:::existing_95:head# 2020-03-11T22:40:01.677 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:ccb3865b:::existing_73:head# 2020-03-11T22:40:01.678 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:cd84e19e:::existing_40:head# 2020-03-11T22:40:01.678 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:d2cd64ce:::existing_37:head# 2020-03-11T22:40:01.679 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:d41a25c9:::existing_22:head# 2020-03-11T22:40:01.679 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:d7f088da:::existing_91:head# 2020-03-11T22:40:01.680 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:40:01.680 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:dcc4ff82:::existing_98:head# 2020-03-11T22:40:01.681 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:dd677bd3:::existing_35:head# 2020-03-11T22:40:01.681 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:dd6d90e7:::existing_46:head# 2020-03-11T22:40:01.682 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:e0e22077:::existing_21:head# 2020-03-11T22:40:01.683 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:e255b710:::existing_55:head# 2020-03-11T22:40:01.683 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:eff1d5db:::existing_14:head# 2020-03-11T22:40:01.684 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:f1b156bf:::existing_23:head# 2020-03-11T22:40:01.684 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:f5ce4199:::existing_34:head# 2020-03-11T22:40:01.685 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:f6809456:::existing_11:head# 2020-03-11T22:40:01.685 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:f699a2a9:::existing_67:head# 2020-03-11T22:40:01.686 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:f9baac43:::existing_25:head# 2020-03-11T22:40:01.686 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:fb76688e:::existing_51:head# 2020-03-11T22:40:01.687 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:fbb92eec:::existing_69:head# 2020-03-11T22:40:01.687 INFO:tasks.workunit.client.0.smithi155.stderr:Export successful 2020-03-11T22:40:01.687 INFO:tasks.workunit.client.0.smithi155.stdout: marking collection for removal 2020-03-11T22:40:01.687 INFO:tasks.workunit.client.0.smithi155.stdout:setting '_remove' omap key 2020-03-11T22:40:01.687 INFO:tasks.workunit.client.0.smithi155.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:40:01.735 INFO:tasks.workunit.client.0.smithi155.stderr:Remove successful 2020-03-11T22:40:02.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:574: TEST_divergent_2: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17464.out 2020-03-11T22:40:02.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:40:02.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:40:02.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:40:02.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:40:02.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:40:02.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17464.out 2020-03-11T22:40:02.658 INFO:tasks.workunit.client.0.smithi155.stdout:get_pg_num_history pg_num_history pg_num_history(e44 pg_nums {1={22=1}} deleted_pools ) 2020-03-11T22:40:02.658 INFO:tasks.workunit.client.0.smithi155.stdout:Importing pgid 1.0 2020-03-11T22:40:02.659 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:40:02.659 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.743 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:40:02.743 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.751 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:40:02.752 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.760 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:091f9f1d:::existing_94:head# 2020-03-11T22:40:02.760 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.768 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:0a865a93:::existing_54:head# 2020-03-11T22:40:02.768 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.776 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:0fed513c:::existing_41:head# 2020-03-11T22:40:02.777 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.785 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:13b9e0b8:::existing_92:head# 2020-03-11T22:40:02.785 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.793 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:169157fa:::existing_81:head# 2020-03-11T22:40:02.793 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.801 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:17731382:::existing_58:head# 2020-03-11T22:40:02.802 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.810 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:18a45a39:::existing_43:head# 2020-03-11T22:40:02.810 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.818 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:1c160f01:::existing_66:head# 2020-03-11T22:40:02.818 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.827 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:1dd6c249:::existing_97:head# 2020-03-11T22:40:02.827 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.835 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:227e0c77:::existing_7:head# 2020-03-11T22:40:02.835 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.843 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:23e1b063:::existing_59:head# 2020-03-11T22:40:02.843 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.881 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:28566edb:::existing_78:head# 2020-03-11T22:40:02.881 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.891 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:2a101182:::existing_20:head# 2020-03-11T22:40:02.891 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.899 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:2d1519da:::existing_96:head# 2020-03-11T22:40:02.899 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.908 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:3078b87a:::existing_86:head# 2020-03-11T22:40:02.908 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.916 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:30bde01b:::existing_15:head# 2020-03-11T22:40:02.916 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.924 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:317a7e19:::existing_56:head# 2020-03-11T22:40:02.924 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.932 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:38b9c7fa:::existing_33:head# 2020-03-11T22:40:02.933 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.941 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:3e16d84b:::existing_48:head# 2020-03-11T22:40:02.941 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.949 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:45fe8203:::existing_75:head# 2020-03-11T22:40:02.950 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.958 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:4e82c9ba:::existing_50:head# 2020-03-11T22:40:02.958 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.966 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:5298dcdb:::existing_57:head# 2020-03-11T22:40:02.966 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.974 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:52ebaa44:::existing_32:head# 2020-03-11T22:40:02.974 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.983 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:53588912:::existing_9:head# 2020-03-11T22:40:02.983 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:02.991 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:5b7e2de6:::existing_84:head# 2020-03-11T22:40:02.991 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.050 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:5da0255c:::existing_13:head# 2020-03-11T22:40:03.051 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.060 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:5da2dab8:::existing_53:head# 2020-03-11T22:40:03.060 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.068 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:5ee3514b:::existing_30:head# 2020-03-11T22:40:03.068 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.077 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:602f83fe:::foo:head# 2020-03-11T22:40:03.077 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.085 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:60426cc7:::existing_62:head# 2020-03-11T22:40:03.085 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.093 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:628918ca:::existing_79:head# 2020-03-11T22:40:03.094 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.103 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:63f8ab2d:::existing_77:head# 2020-03-11T22:40:03.103 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.110 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:67e1780f:::existing_27:head# 2020-03-11T22:40:03.110 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.118 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:68143ca1:::existing_71:head# 2020-03-11T22:40:03.119 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.127 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:6bac9509:::existing_29:head# 2020-03-11T22:40:03.127 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.135 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:6bbec798:::existing_45:head# 2020-03-11T22:40:03.135 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.144 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:6cb23a40:::existing_65:head# 2020-03-11T22:40:03.144 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.152 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:7055ec5e:::existing_83:head# 2020-03-11T22:40:03.152 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.195 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:70a54e2b:::existing_87:head# 2020-03-11T22:40:03.195 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.202 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:750ace21:::existing_19:head# 2020-03-11T22:40:03.202 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.210 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:78623b1d:::existing_6:head# 2020-03-11T22:40:03.210 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.219 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:79881152:::existing_63:head# 2020-03-11T22:40:03.219 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.227 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:7a80c0d5:::existing_99:head# 2020-03-11T22:40:03.227 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.235 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:81490677:::existing_76:head# 2020-03-11T22:40:03.236 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.244 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:85ec8e5d:::existing_85:head# 2020-03-11T22:40:03.244 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.252 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:8e2abbf0:::existing_49:head# 2020-03-11T22:40:03.252 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.260 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:8ee0fe0f:::existing_52:head# 2020-03-11T22:40:03.260 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.269 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:923c5d4d:::existing_5:head# 2020-03-11T22:40:03.269 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.277 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:92731e71:::existing_89:head# 2020-03-11T22:40:03.278 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.286 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:928205cf:::existing_82:head# 2020-03-11T22:40:03.287 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.294 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:93be2267:::existing_24:head# 2020-03-11T22:40:03.294 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.302 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:99ceda2c:::existing_44:head# 2020-03-11T22:40:03.302 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.340 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:9dd917ee:::existing_26:head# 2020-03-11T22:40:03.340 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.377 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:a28805a6:::existing_72:head# 2020-03-11T22:40:03.378 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.386 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:a294a8f6:::existing_16:head# 2020-03-11T22:40:03.386 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.394 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:a2f5473a:::existing_42:head# 2020-03-11T22:40:03.394 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.414 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:a504a533:::existing_90:head# 2020-03-11T22:40:03.415 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.415 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:a691337e:::existing_61:head# 2020-03-11T22:40:03.415 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.420 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:a6e8d7ac:::existing_28:head# 2020-03-11T22:40:03.420 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.427 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:a9fde86e:::existing_18:head# 2020-03-11T22:40:03.427 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.436 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:aa309faa:::existing_68:head# 2020-03-11T22:40:03.436 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.444 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:ae40b0ed:::existing_31:head# 2020-03-11T22:40:03.444 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.480 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:b279b1fe:::existing_10:head# 2020-03-11T22:40:03.480 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.519 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:b343e277:::existing_88:head# 2020-03-11T22:40:03.519 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.531 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:b7130e90:::existing_80:head# 2020-03-11T22:40:03.532 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.582 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:bc6f7309:::existing_70:head# 2020-03-11T22:40:03.582 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.598 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:be8489b7:::existing_38:head# 2020-03-11T22:40:03.599 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.602 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:beb4e5e3:::existing_47:head# 2020-03-11T22:40:03.603 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.611 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:bebec877:::existing_74:head# 2020-03-11T22:40:03.611 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.619 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:c04347d8:::existing_60:head# 2020-03-11T22:40:03.619 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.628 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:c156033e:::existing_100:head# 2020-03-11T22:40:03.628 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.636 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:c19f5a8e:::existing_8:head# 2020-03-11T22:40:03.636 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.644 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:c3e6ebe4:::existing_17:head# 2020-03-11T22:40:03.644 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.653 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:c6c5a7c2:::existing_39:head# 2020-03-11T22:40:03.653 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.681 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:cc1d0abf:::existing_95:head# 2020-03-11T22:40:03.681 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.694 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:ccb3865b:::existing_73:head# 2020-03-11T22:40:03.695 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.703 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:cd84e19e:::existing_40:head# 2020-03-11T22:40:03.703 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.711 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:d2cd64ce:::existing_37:head# 2020-03-11T22:40:03.711 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.719 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:d41a25c9:::existing_22:head# 2020-03-11T22:40:03.720 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.773 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:d7f088da:::existing_91:head# 2020-03-11T22:40:03.774 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.786 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:d9cbd1d6:::existing_64:head# 2020-03-11T22:40:03.786 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.794 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:dcc4ff82:::existing_98:head# 2020-03-11T22:40:03.795 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.803 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:dd677bd3:::existing_35:head# 2020-03-11T22:40:03.803 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.811 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:dd6d90e7:::existing_46:head# 2020-03-11T22:40:03.811 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.819 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:e0e22077:::existing_21:head# 2020-03-11T22:40:03.820 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.828 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:e255b710:::existing_55:head# 2020-03-11T22:40:03.828 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.836 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:eff1d5db:::existing_14:head# 2020-03-11T22:40:03.836 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.844 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:f1b156bf:::existing_23:head# 2020-03-11T22:40:03.845 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.853 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:f5ce4199:::existing_34:head# 2020-03-11T22:40:03.853 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.861 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:f6809456:::existing_11:head# 2020-03-11T22:40:03.861 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.870 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:f699a2a9:::existing_67:head# 2020-03-11T22:40:03.870 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.878 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:f9baac43:::existing_25:head# 2020-03-11T22:40:03.878 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.886 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:fb76688e:::existing_51:head# 2020-03-11T22:40:03.886 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.924 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:fbb92eec:::existing_69:head# 2020-03-11T22:40:03.924 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:40:03.936 INFO:tasks.workunit.client.0.smithi155.stdout:write_pg epoch 44 info 1.0( v 44'102 lc 0'0 (28'90,44'102] local-lis/les=43/44 n=101 ec=22/22 lis/c=43/22 les/c/f=44/23/0 sis=43 pruub=8.645268676s) 2020-03-11T22:40:03.953 INFO:tasks.workunit.client.0.smithi155.stdout:Import successful 2020-03-11T22:40:04.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:576: TEST_divergent_2: echo 'reviving divergent 1' 2020-03-11T22:40:04.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:577: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:04.478 INFO:tasks.workunit.client.0.smithi155.stdout:reviving divergent 1 2020-03-11T22:40:04.808 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:04.808 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 4 4 0 0 47947 0 0 11 11 active+recovering+degraded+wait 2020-03-11T22:39:51.806979+0000 39'101 44:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:40:04.808 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:40:04.808 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:04.809 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:40:04.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:578: TEST_divergent_2: activate_osd td/divergent-priors 1 2020-03-11T22:40:04.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:40:04.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:40:04.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:40:04.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:40:04.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:40:04.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:04.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:04.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:04.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:04.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:40:04.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:40:04.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:04.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:40:04.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:04.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:40:04.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:04.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:04.823 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:04.823 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:04.823 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:40:04.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:40:04.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:40:04.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:04.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:04.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:04.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:04.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:04.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:04.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:40:04.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:40:04.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:40:04.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:40:04.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:04.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:40:04.826 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:40:04.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:40:04.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:40:04.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:40:04.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:40:04.847 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:04.844+0000 7f085f788ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:04.849 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:04.846+0000 7f085f788ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:04.849 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:04.847+0000 7f085f788ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:05.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:40:05.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:05.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:40:05.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:05.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:05.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:05.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:05.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:05.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:05.171 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:40:05.365 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:05.362+0000 7f085f788ec0 -1 Falling back to public interface 2020-03-11T22:40:05.517 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6818/33107,v1:127.0.0.1:6819/33107] [v2:127.0.0.1:6820/33107,v1:127.0.0.1:6821/33107] exists,up 1b163036-a7f8-46e5-b80e-ffab673e9327 2020-03-11T22:40:05.517 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:40:05.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:05.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:05.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:05.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:579: TEST_divergent_2: wait_for_osd up 1 2020-03-11T22:40:05.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:05.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:40:05.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:05.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:05.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:05.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:05.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:05.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:05.862 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 43 up_thru 43 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6818/33107,v1:127.0.0.1:6819/33107] [v2:127.0.0.1:6820/33107,v1:127.0.0.1:6821/33107] exists,up 1b163036-a7f8-46e5-b80e-ffab673e9327 2020-03-11T22:40:05.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:05.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:05.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:05.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:581: TEST_divergent_2: sleep 20 2020-03-11T22:40:06.149 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:06.145+0000 7f085f788ec0 -1 osd.1 44 log_to_monitors {default=true} 2020-03-11T22:40:25.864 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: get_asok_path osd.1 2020-03-11T22:40:25.865 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:40:25.865 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:40:25.865 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:25.865 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:25.865 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:40:25.866 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.1.asok 2020-03-11T22:40:25.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: CEPH_ARGS= 2020-03-11T22:40:25.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:582: TEST_divergent_2: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:40:25.965 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:40:25.965 INFO:tasks.workunit.client.0.smithi155.stdout: "ops": [], 2020-03-11T22:40:25.965 INFO:tasks.workunit.client.0.smithi155.stdout: "num_ops": 0 2020-03-11T22:40:25.965 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:40:25.971 INFO:tasks.workunit.client.0.smithi155.stdout:allowing recovery 2020-03-11T22:40:25.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:584: TEST_divergent_2: echo 'allowing recovery' 2020-03-11T22:40:25.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:585: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:26.321 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:26.321 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:40:07.446648+0000 44'102 47:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:40:26.322 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:40:26.322 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:26.322 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:40:26.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:40:26.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:40:26.451 INFO:tasks.workunit.client.0.smithi155.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:40:26.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:40:26.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:40:26.577 INFO:tasks.workunit.client.0.smithi155.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:40:26.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:587: TEST_divergent_2: for i in $osds 2020-03-11T22:40:26.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:589: TEST_divergent_2: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:40:26.706 INFO:tasks.workunit.client.0.smithi155.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:40:26.711 INFO:tasks.workunit.client.0.smithi155.stdout:reading divergent objects 2020-03-11T22:40:26.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:592: TEST_divergent_2: echo 'reading divergent objects' 2020-03-11T22:40:26.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:593: TEST_divergent_2: ceph pg dump pgs 2020-03-11T22:40:27.030 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:40:27.030 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 0 0 0 0 48430 0 0 12 12 active+clean 2020-03-11T22:40:07.446648+0000 44'102 47:148 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:37:50.816593+0000 0'0 2020-03-11T22:37:50.816593+0000 0 2020-03-11T22:40:27.031 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:40:27.031 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:40:27.032 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:40:27.040 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: expr 2 + 2 2020-03-11T22:40:27.041 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: seq 1 4 2020-03-11T22:40:27.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:40:27.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:40:27.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:40:27.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:40:27.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:40:27.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:40:27.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:594: TEST_divergent_2: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:40:27.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:596: TEST_divergent_2: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:40:27.155 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: seq 1 2 2020-03-11T22:40:27.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:40:27.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:40:27.182 INFO:tasks.workunit.client.0.smithi155.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:40:27.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:598: TEST_divergent_2: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:40:27.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:600: TEST_divergent_2: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:40:27.210 INFO:tasks.workunit.client.0.smithi155.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:40:27.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:602: TEST_divergent_2: rm -f td/divergent-priors/existing 2020-03-11T22:40:27.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:40:27.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:604: TEST_divergent_2: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:40:27.216 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4543.0:1 2020-03-11T22:38:13.600989+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:40:27.216 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 20 _merge_object_divergent_entries: keeping 28'106 (28'4) delete 1:70b3bf12:::existing_4:head by client.4543.0:1 2020-03-11T22:38:13.600989+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:40:27.216 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:40:27.216 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 28'4 first_divergent_update: 28'106 last_divergent_update: 28'106 2020-03-11T22:40:27.216 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: more recent entry found: 39'101 (28'4) modify 1:70b3bf12:::existing_4:head by client.4654.0:1 2020-03-11T22:39:07.496492+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:40:27.217 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 28'102 (0'0) modify 1:75ac5091:::newobject_2:head by client.4531.0:1 2020-03-11T22:38:13.581921+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:40:27.217 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 20 _merge_object_divergent_entries: keeping 28'102 (0'0) modify 1:75ac5091:::newobject_2:head by client.4531.0:1 2020-03-11T22:38:13.581921+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:40:27.217 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:40:27.217 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 28'102 last_divergent_update: 28'102 2020-03-11T22:40:27.217 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:40:27.217 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:40:27.218 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:38:13.599859+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:40:27.218 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 20 _merge_object_divergent_entries: keeping 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:38:13.599859+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:40:27.218 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:40:27.218 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 28'3 first_divergent_update: 28'105 last_divergent_update: 28'105 2020-03-11T22:40:27.218 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:40:27.218 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:40:27.218 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 28'106 2020-03-11T22:40:27.218 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 28'105 (28'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:38:13.599859+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:40:27.219 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:40:27.219 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 28'3 <= info.log_tail 28'90 2020-03-11T22:40:27.219 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 28'104 (28'2) modify 1:97db2a0f:::existing_2:head by client.4539.0:1 2020-03-11T22:38:13.597526+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:27.219 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 20 _merge_object_divergent_entries: keeping 28'104 (28'2) modify 1:97db2a0f:::existing_2:head by client.4539.0:1 2020-03-11T22:38:13.597526+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:27.219 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:40:27.219 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 28'2 first_divergent_update: 28'104 last_divergent_update: 28'104 2020-03-11T22:40:27.219 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:40:27.220 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:40:27.220 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 28'106 2020-03-11T22:40:27.220 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 28'104 (28'2) modify 1:97db2a0f:::existing_2:head by client.4539.0:1 2020-03-11T22:38:13.597526+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:27.220 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:40:27.220 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 28'2 <= info.log_tail 28'90 2020-03-11T22:40:27.220 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 28'103 (28'1) modify 1:d858c035:::existing_1:head by client.4536.0:1 2020-03-11T22:38:13.595028+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:27.220 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 20 _merge_object_divergent_entries: keeping 28'103 (28'1) modify 1:d858c035:::existing_1:head by client.4536.0:1 2020-03-11T22:38:13.595028+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:27.221 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:40:27.221 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 28'1 first_divergent_update: 28'103 last_divergent_update: 28'103 2020-03-11T22:40:27.221 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:40:27.222 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:40:27.222 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 28'106 2020-03-11T22:40:27.222 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 28'103 (28'1) modify 1:d858c035:::existing_1:head by client.4536.0:1 2020-03-11T22:38:13.595028+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:40:27.222 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:40:27.222 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 28'1 <= info.log_tail 28'90 2020-03-11T22:40:27.222 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 28'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4530.0:1 2020-03-11T22:38:13.580495+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:40:27.222 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 20 _merge_object_divergent_entries: keeping 28'101 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4530.0:1 2020-03-11T22:38:13.580495+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:40:27.223 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:40:27.223 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 28'101 last_divergent_update: 28'101 2020-03-11T22:40:27.223 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:40:27.223 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:39:51.633+0000 7fc0aa5d7700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:40:27.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:40:27.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:606: TEST_divergent_2: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:40:27.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: find td/divergent-priors -name '*osd*log' 2020-03-11T22:40:27.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:612: TEST_divergent_2: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:40:27.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:617: TEST_divergent_2: echo success 2020-03-11T22:40:27.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:619: TEST_divergent_2: rm td/divergent-priors/td/divergent-priors/exp.17464.out 2020-03-11T22:40:27.225 INFO:tasks.workunit.client.0.smithi155.stdout:success 2020-03-11T22:40:27.225 INFO:tasks.workunit.client.0.smithi155.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17464.out': No such file or directory 2020-03-11T22:40:27.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:621: TEST_divergent_2: delete_pool test 2020-03-11T22:40:27.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:40:27.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:40:27.700 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test' does not exist 2020-03-11T22:40:27.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:622: TEST_divergent_2: kill_daemons td/divergent-priors 2020-03-11T22:40:27.708 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:27.709 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:27.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:27.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:27.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:27.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:27.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:40:27.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:40:27.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:40:27.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:40:27.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:27.818 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:27.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:27.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:27.818 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:27.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:27.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:40:27.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:40:27.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:40:27.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:40:27.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:40:27.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:40:27.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:40:27.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:40:27.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:40:27.826 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:40:27.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:40:27.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:40:27.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:40:27.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:40:27.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:27.840 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:40:27.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17464 2020-03-11T22:40:27.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:40:27.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:40:27.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:40:27.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:40:27.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:40:27.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:40:27.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:40:27.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:40:27.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:40:27.843 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:40:27.843 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:40:27.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:40:27.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:40:27.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:40:27.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:40:27.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:40:27.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:40:27.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:40:27.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:40:27.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:40:27.846 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:40:27.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:40:27.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:40:27.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:40:27.847 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:40:27.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:40:27.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:40:27.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:40:27.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:40:27.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:27.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:40:27.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17464 2020-03-11T22:40:27.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:40:27.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:40:27.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:40:27.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:40:27.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:27.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:40:27.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17464 2020-03-11T22:40:27.854 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:40:27.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:40:27.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:40:27.854 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:40:27.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_3 td/divergent-priors 2020-03-11T22:40:27.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:628: TEST_divergent_3: local dir=td/divergent-priors 2020-03-11T22:40:27.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:631: TEST_divergent_3: local dummyfile=/etc/fstab 2020-03-11T22:40:27.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:632: TEST_divergent_3: local dummyfile2=/etc/resolv.conf 2020-03-11T22:40:27.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:634: TEST_divergent_3: local num_osds=3 2020-03-11T22:40:27.855 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: expr 3 - 1 2020-03-11T22:40:27.856 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: seq 0 2 2020-03-11T22:40:27.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:635: TEST_divergent_3: local 'osds=0 2020-03-11T22:40:27.857 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:40:27.857 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:40:27.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:636: TEST_divergent_3: run_mon td/divergent-priors a 2020-03-11T22:40:27.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:40:27.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:40:27.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:40:27.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:40:27.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:40:27.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:40:28.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:40:28.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:28.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:28.154 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:28.155 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:28.155 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:40:28.155 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:40:28.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:40:28.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:40:28.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:40:28.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:40:28.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:40:28.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:40:28.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:40:28.308 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:40:28.308 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:40:28.309 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:40:28.309 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:28.309 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:28.309 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:40:28.310 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-mon.a.asok 2020-03-11T22:40:28.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:40:28.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17464/ceph-mon.a.asok config get fsid 2020-03-11T22:40:28.395 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:40:28.395 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:40:28.395 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:40:28.395 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:40:28.395 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:40:28.396 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:40:28.396 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:40:28.396 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:40:28.396 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:40:28.396 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:28.397 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:40:28.397 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-mon.a.asok 2020-03-11T22:40:28.397 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:40:28.397 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17464/ceph-mon.a.asok config get mon_host 2020-03-11T22:40:28.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:637: TEST_divergent_3: run_mgr td/divergent-priors x 2020-03-11T22:40:28.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:40:28.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:40:28.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:40:28.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:40:28.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:40:28.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:40:28.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:40:28.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:28.972 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:28.972 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:28.973 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:28.973 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:40:28.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:40:28.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:40:28.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:40:29.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:40:29.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 0 2020-03-11T22:40:29.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:40:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:40:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:40:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:40:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:40:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:29.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:29.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:29.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:40:29.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:40:29.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:29.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:40:29.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:29.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:40:29.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:29.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:29.004 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:29.004 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:29.004 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:40:29.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:40:29.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:40:29.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:29.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:40:29.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:40:29.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:29.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:29.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:29.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:29.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:29.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:40:29.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:40:29.006 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:40:29.006 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:40:29.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=53c048ef-1ae8-4c15-b86c-b15481d0a054 2020-03-11T22:40:29.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 53c048ef-1ae8-4c15-b86c-b15481d0a054' 2020-03-11T22:40:29.007 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:40:29.007 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 53c048ef-1ae8-4c15-b86c-b15481d0a054 2020-03-11T22:40:29.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDdaGlelqO+ARAA5+7T3CZEaPxTDVn3bDk2Zg== 2020-03-11T22:40:29.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDdaGlelqO+ARAA5+7T3CZEaPxTDVn3bDk2Zg=="}' 2020-03-11T22:40:29.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 53c048ef-1ae8-4c15-b86c-b15481d0a054 -i td/divergent-priors/0/new.json 2020-03-11T22:40:29.353 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:40:29.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:40:29.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDdaGlelqO+ARAA5+7T3CZEaPxTDVn3bDk2Zg== --osd-uuid 53c048ef-1ae8-4c15-b86c-b15481d0a054 2020-03-11T22:40:29.392 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:29.390+0000 7f4aa1d6bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:29.394 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:29.392+0000 7f4aa1d6bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:29.396 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:29.393+0000 7f4aa1d6bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:29.396 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:29.393+0000 7f4aa1d6bec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:40:29.397 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:29.394+0000 7f4aa1d6bec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:40:29.397 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:29.394+0000 7f4aa1d6bec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:40:29.455 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:29.453+0000 7f4aa1d6bec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:40:32.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:40:32.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:40:32.470 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2020-03-11T22:40:32.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:40:32.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:40:32.917 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:40:32.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:40:32.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:40:32.918 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:32.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:40:32.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:40:32.944 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:32.941+0000 7f47750d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:32.946 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:32.943+0000 7f47750d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:32.947 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:32.944+0000 7f47750d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:33.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:40:33.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:33.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:40:33.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:33.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:33.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:33.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:33.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:33.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:33.270 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:40:33.463 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:33.460+0000 7f47750d4ec0 -1 Falling back to public interface 2020-03-11T22:40:33.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:34.244 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:34.242+0000 7f47750d4ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:40:34.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:34.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:34.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:34.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:34.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:40:34.619 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:40:34.971 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/34605,v1:127.0.0.1:6803/34605] [v2:127.0.0.1:6804/34605,v1:127.0.0.1:6805/34605] exists,up 53c048ef-1ae8-4c15-b86c-b15481d0a054 2020-03-11T22:40:34.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:34.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:34.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:34.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:40:34.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 1 2020-03-11T22:40:34.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:40:34.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:40:34.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:40:34.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:40:34.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:40:34.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:34.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:34.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:34.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:34.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:40:34.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:40:34.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:34.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:40:34.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:34.973 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:40:34.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:34.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:34.974 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:34.974 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:34.974 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:40:34.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:40:34.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:40:34.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:34.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:40:34.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:40:34.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:34.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:34.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:34.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:34.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:34.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:40:34.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:40:34.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:40:34.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:40:34.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=ce1120e9-d62c-42de-b0b2-a04b6602228c 2020-03-11T22:40:34.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 ce1120e9-d62c-42de-b0b2-a04b6602228c' 2020-03-11T22:40:34.978 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:40:34.979 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 ce1120e9-d62c-42de-b0b2-a04b6602228c 2020-03-11T22:40:34.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDiaGleYX/jOhAAORym9n1QExuYAdbvaLVpUg== 2020-03-11T22:40:34.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDiaGleYX/jOhAAORym9n1QExuYAdbvaLVpUg=="}' 2020-03-11T22:40:34.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new ce1120e9-d62c-42de-b0b2-a04b6602228c -i td/divergent-priors/1/new.json 2020-03-11T22:40:35.477 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:40:35.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:40:35.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDiaGleYX/jOhAAORym9n1QExuYAdbvaLVpUg== --osd-uuid ce1120e9-d62c-42de-b0b2-a04b6602228c 2020-03-11T22:40:35.515 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:35.512+0000 7ff3eb5d8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:35.516 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:35.514+0000 7ff3eb5d8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:35.518 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:35.516+0000 7ff3eb5d8ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:35.519 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:35.516+0000 7ff3eb5d8ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:40:35.519 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:35.517+0000 7ff3eb5d8ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:40:35.519 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:35.517+0000 7ff3eb5d8ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:40:35.601 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:35.598+0000 7ff3eb5d8ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:40:38.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:40:38.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:40:38.678 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2020-03-11T22:40:38.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:40:38.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:40:39.137 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:40:39.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:40:39.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:40:39.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:39.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:40:39.138 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:40:39.163 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:39.160+0000 7f8cbfac6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:39.164 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:39.162+0000 7f8cbfac6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:39.166 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:39.163+0000 7f8cbfac6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:39.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:40:39.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:39.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:40:39.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:39.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:39.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:39.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:39.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:39.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:39.482 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:40:39.682 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:39.680+0000 7f8cbfac6ec0 -1 Falling back to public interface 2020-03-11T22:40:39.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:40.475 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:40.473+0000 7f8cbfac6ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:40:40.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:40.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:40.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:40.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:40.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:40:40.828 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:40:41.187 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/34960,v1:127.0.0.1:6811/34960] [v2:127.0.0.1:6812/34960,v1:127.0.0.1:6813/34960] exists,up ce1120e9-d62c-42de-b0b2-a04b6602228c 2020-03-11T22:40:41.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:41.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:41.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:41.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:638: TEST_divergent_3: for i in $osds 2020-03-11T22:40:41.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:640: TEST_divergent_3: run_osd td/divergent-priors 2 2020-03-11T22:40:41.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:40:41.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:40:41.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:40:41.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:40:41.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:40:41.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:40:41.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:40:41.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:40:41.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:40:41.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:40:41.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:40:41.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:40:41.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:40:41.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:40:41.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:40:41.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:40:41.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:40:41.191 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:40:41.191 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:40:41.191 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:40:41.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:40:41.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:40:41.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:40:41.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:40:41.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:40:41.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:40:41.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:40:41.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:40:41.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:40:41.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:41.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:40:41.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:40:41.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:40:41.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:40:41.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=8a4738de-77ce-4790-b3b3-87d310c2cbe9 2020-03-11T22:40:41.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 8a4738de-77ce-4790-b3b3-87d310c2cbe9' 2020-03-11T22:40:41.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:40:41.195 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 8a4738de-77ce-4790-b3b3-87d310c2cbe9 2020-03-11T22:40:41.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDpaGlebeciDBAALe1JQzaldKM0VVR5KjG1zA== 2020-03-11T22:40:41.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDpaGlebeciDBAALe1JQzaldKM0VVR5KjG1zA=="}' 2020-03-11T22:40:41.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 8a4738de-77ce-4790-b3b3-87d310c2cbe9 -i td/divergent-priors/2/new.json 2020-03-11T22:40:41.758 INFO:tasks.workunit.client.0.smithi155.stdout:2 2020-03-11T22:40:41.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:40:41.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDpaGlebeciDBAALe1JQzaldKM0VVR5KjG1zA== --osd-uuid 8a4738de-77ce-4790-b3b3-87d310c2cbe9 2020-03-11T22:40:41.796 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:41.794+0000 7f113a82aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:41.816 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:41.813+0000 7f113a82aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:41.817 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:41.814+0000 7f113a82aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:41.817 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:41.814+0000 7f113a82aec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:40:41.817 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:41.814+0000 7f113a82aec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:40:41.817 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:41.814+0000 7f113a82aec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:40:41.882 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:41.880+0000 7f113a82aec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:40:45.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:40:45.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:40:45.127 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2020-03-11T22:40:45.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:40:45.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:40:45.580 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:40:45.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:40:45.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:40:45.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:40:45.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:40:45.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:40:45.607 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:45.604+0000 7f41ec1d5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:45.608 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:45.606+0000 7f41ec1d5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:45.611 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:45.608+0000 7f41ec1d5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:40:45.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:40:45.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:40:45.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:40:45.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:40:45.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:40:45.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:45.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:40:45.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:45.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:45.927 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:40:46.126 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:46.123+0000 7f41ec1d5ec0 -1 Falling back to public interface 2020-03-11T22:40:46.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:40:46.866 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:40:46.863+0000 7f41ec1d5ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:40:47.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:40:47.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:40:47.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:40:47.274 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:40:47.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:40:47.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:40:47.647 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/35315,v1:127.0.0.1:6819/35315] [v2:127.0.0.1:6820/35315,v1:127.0.0.1:6821/35315] exists,up 8a4738de-77ce-4790-b3b3-87d310c2cbe9 2020-03-11T22:40:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:40:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:40:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:40:47.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:643: TEST_divergent_3: ceph osd set noout 2020-03-11T22:40:48.318 INFO:tasks.workunit.client.0.smithi155.stderr:noout is set 2020-03-11T22:40:48.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:644: TEST_divergent_3: ceph osd set noin 2020-03-11T22:40:48.935 INFO:tasks.workunit.client.0.smithi155.stderr:noin is set 2020-03-11T22:40:48.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:645: TEST_divergent_3: ceph osd set nodown 2020-03-11T22:40:49.578 INFO:tasks.workunit.client.0.smithi155.stderr:nodown is set 2020-03-11T22:40:49.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:646: TEST_divergent_3: create_pool test 1 1 2020-03-11T22:40:49.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:40:50.154 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test' already exists 2020-03-11T22:40:50.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:40:51.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:647: TEST_divergent_3: ceph osd pool set test size 3 2020-03-11T22:40:51.782 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 size to 3 2020-03-11T22:40:51.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:648: TEST_divergent_3: ceph osd pool set test min_size 2 2020-03-11T22:40:52.476 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 min_size to 2 2020-03-11T22:40:52.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:651: TEST_divergent_3: ceph osd pool set test pg_autoscale_mode on 2020-03-11T22:40:53.094 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 pg_autoscale_mode to on 2020-03-11T22:40:53.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:653: TEST_divergent_3: flush_pg_stats 2020-03-11T22:40:53.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:53.102 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:53.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:53.452 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:40:53.452 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:40:53.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:53.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:53.453 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:53.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:40:53.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:40:53.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:53.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:53.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705669 2020-03-11T22:40:53.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-38654705669' 2020-03-11T22:40:53.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:53.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:53.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542148 2020-03-11T22:40:53.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-38654705669 2-60129542148' 2020-03-11T22:40:53.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:53.841 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:40:53.841 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:53.841 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:53.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:40:53.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:53.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:40:53.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:40:53.843 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:53.843 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:40:54.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:40:54.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:40:55.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:40:55.200 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:55.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:40:55.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:55.553 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705669 2020-03-11T22:40:55.553 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:55.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:55.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705669 2020-03-11T22:40:55.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:55.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705669 2020-03-11T22:40:55.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705669' 2020-03-11T22:40:55.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:55.556 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705669 2020-03-11T22:40:55.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705669 -lt 38654705669 2020-03-11T22:40:55.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:55.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542148 2020-03-11T22:40:55.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:55.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:55.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542148 2020-03-11T22:40:55.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:55.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542148 2020-03-11T22:40:55.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2020-03-11T22:40:55.899 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542148 2020-03-11T22:40:55.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:56.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542148 -lt 60129542148 2020-03-11T22:40:56.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:654: TEST_divergent_3: wait_for_clean 2020-03-11T22:40:56.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:40:56.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:40:56.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:40:56.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:40:56.268 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:40:56.268 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:40:56.268 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:40:56.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:40:56.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:40:56.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:40:56.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:40:56.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:40:56.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:40:56.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:40:56.361 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:40:56.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:40:56.729 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:40:56.729 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:40:56.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:40:56.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:56.730 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:40:56.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836489 2020-03-11T22:40:56.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489' 2020-03-11T22:40:56.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:56.859 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:40:56.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705672 2020-03-11T22:40:56.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-38654705672' 2020-03-11T22:40:56.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:40:56.988 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:40:57.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542150 2020-03-11T22:40:57.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836489 1-38654705672 2-60129542150' 2020-03-11T22:40:57.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:57.118 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836489 2020-03-11T22:40:57.118 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:57.118 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:40:57.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836489 2020-03-11T22:40:57.119 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:57.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836489 2020-03-11T22:40:57.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836489' 2020-03-11T22:40:57.120 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836489 2020-03-11T22:40:57.120 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:57.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836489 2020-03-11T22:40:57.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:40:58.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:40:58.460 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:40:58.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836489 -lt 21474836489 2020-03-11T22:40:58.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:58.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705672 2020-03-11T22:40:58.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:58.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:40:58.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705672 2020-03-11T22:40:58.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:58.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705672 2020-03-11T22:40:58.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705672' 2020-03-11T22:40:58.813 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705672 2020-03-11T22:40:58.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:40:59.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705672 -lt 38654705672 2020-03-11T22:40:59.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:40:59.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542150 2020-03-11T22:40:59.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:40:59.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:40:59.178 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542150 2020-03-11T22:40:59.178 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:40:59.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542150 2020-03-11T22:40:59.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2020-03-11T22:40:59.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:40:59.180 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542150 2020-03-11T22:40:59.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542151 -lt 60129542150 2020-03-11T22:40:59.526 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:40:59.526 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:40:59.526 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:40:59.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:40:59.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:40:59.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:40:59.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:40:59.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:40:59.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:40:59.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:40:59.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:00.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:41:00.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:00.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:00.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:00.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:41:00.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:41:00.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:41:00.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: ceph pg dump pgs --format=json 2020-03-11T22:41:00.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: jq '.pg_stats[0].up_primary' 2020-03-11T22:41:01.192 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:41:01.200 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:657: TEST_divergent_3: local divergent=1 2020-03-11T22:41:01.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:658: TEST_divergent_3: echo 'primary and soon to be divergent is 1' 2020-03-11T22:41:01.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:659: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:01.201 INFO:tasks.workunit.client.0.smithi155.stdout:primary and soon to be divergent is 1 2020-03-11T22:41:01.522 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:01.522 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:40:50.896158+0000 0'0 26:13 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:41:01.522 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:41:01.523 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:01.524 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:41:01.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:660: TEST_divergent_3: local non_divergent= 2020-03-11T22:41:01.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:41:01.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 0 = 1 ']' 2020-03-11T22:41:01.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0' 2020-03-11T22:41:01.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:41:01.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 1 = 1 ']' 2020-03-11T22:41:01.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:664: TEST_divergent_3: continue 2020-03-11T22:41:01.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:661: TEST_divergent_3: for i in $osds 2020-03-11T22:41:01.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:663: TEST_divergent_3: '[' 2 = 1 ']' 2020-03-11T22:41:01.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:666: TEST_divergent_3: non_divergent=' 0 2' 2020-03-11T22:41:01.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:669: TEST_divergent_3: echo 'writing initial objects' 2020-03-11T22:41:01.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: seq 1 100 2020-03-11T22:41:01.534 INFO:tasks.workunit.client.0.smithi155.stdout:writing initial objects 2020-03-11T22:41:01.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:01.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_1 /etc/fstab 2020-03-11T22:41:01.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:01.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_2 /etc/fstab 2020-03-11T22:41:01.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:01.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_3 /etc/fstab 2020-03-11T22:41:01.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:01.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_4 /etc/fstab 2020-03-11T22:41:01.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:01.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_5 /etc/fstab 2020-03-11T22:41:01.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:01.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_6 /etc/fstab 2020-03-11T22:41:01.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:01.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_7 /etc/fstab 2020-03-11T22:41:01.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:01.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_8 /etc/fstab 2020-03-11T22:41:02.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_9 /etc/fstab 2020-03-11T22:41:02.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_10 /etc/fstab 2020-03-11T22:41:02.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_11 /etc/fstab 2020-03-11T22:41:02.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_12 /etc/fstab 2020-03-11T22:41:02.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_13 /etc/fstab 2020-03-11T22:41:02.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_14 /etc/fstab 2020-03-11T22:41:02.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_15 /etc/fstab 2020-03-11T22:41:02.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_16 /etc/fstab 2020-03-11T22:41:02.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_17 /etc/fstab 2020-03-11T22:41:02.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_18 /etc/fstab 2020-03-11T22:41:02.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_19 /etc/fstab 2020-03-11T22:41:02.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_20 /etc/fstab 2020-03-11T22:41:02.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_21 /etc/fstab 2020-03-11T22:41:02.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_22 /etc/fstab 2020-03-11T22:41:02.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_23 /etc/fstab 2020-03-11T22:41:02.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:02.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_24 /etc/fstab 2020-03-11T22:41:03.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_25 /etc/fstab 2020-03-11T22:41:03.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_26 /etc/fstab 2020-03-11T22:41:03.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_27 /etc/fstab 2020-03-11T22:41:03.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_28 /etc/fstab 2020-03-11T22:41:03.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_29 /etc/fstab 2020-03-11T22:41:03.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_30 /etc/fstab 2020-03-11T22:41:03.468 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_31 /etc/fstab 2020-03-11T22:41:03.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_32 /etc/fstab 2020-03-11T22:41:03.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_33 /etc/fstab 2020-03-11T22:41:03.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_34 /etc/fstab 2020-03-11T22:41:03.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_35 /etc/fstab 2020-03-11T22:41:03.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_36 /etc/fstab 2020-03-11T22:41:03.848 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_37 /etc/fstab 2020-03-11T22:41:03.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:03.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_38 /etc/fstab 2020-03-11T22:41:04.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_39 /etc/fstab 2020-03-11T22:41:04.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_40 /etc/fstab 2020-03-11T22:41:04.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_41 /etc/fstab 2020-03-11T22:41:04.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_42 /etc/fstab 2020-03-11T22:41:04.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_43 /etc/fstab 2020-03-11T22:41:04.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_44 /etc/fstab 2020-03-11T22:41:04.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_45 /etc/fstab 2020-03-11T22:41:04.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_46 /etc/fstab 2020-03-11T22:41:04.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_47 /etc/fstab 2020-03-11T22:41:04.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_48 /etc/fstab 2020-03-11T22:41:04.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_49 /etc/fstab 2020-03-11T22:41:04.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_50 /etc/fstab 2020-03-11T22:41:04.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_51 /etc/fstab 2020-03-11T22:41:04.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_52 /etc/fstab 2020-03-11T22:41:04.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:04.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_53 /etc/fstab 2020-03-11T22:41:05.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_54 /etc/fstab 2020-03-11T22:41:05.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_55 /etc/fstab 2020-03-11T22:41:05.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_56 /etc/fstab 2020-03-11T22:41:05.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_57 /etc/fstab 2020-03-11T22:41:05.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.258 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_58 /etc/fstab 2020-03-11T22:41:05.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_59 /etc/fstab 2020-03-11T22:41:05.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_60 /etc/fstab 2020-03-11T22:41:05.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_61 /etc/fstab 2020-03-11T22:41:05.522 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_62 /etc/fstab 2020-03-11T22:41:05.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_63 /etc/fstab 2020-03-11T22:41:05.607 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_64 /etc/fstab 2020-03-11T22:41:05.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_65 /etc/fstab 2020-03-11T22:41:05.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_66 /etc/fstab 2020-03-11T22:41:05.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_67 /etc/fstab 2020-03-11T22:41:05.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_68 /etc/fstab 2020-03-11T22:41:05.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_69 /etc/fstab 2020-03-11T22:41:05.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:05.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_70 /etc/fstab 2020-03-11T22:41:06.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_71 /etc/fstab 2020-03-11T22:41:06.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_72 /etc/fstab 2020-03-11T22:41:06.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_73 /etc/fstab 2020-03-11T22:41:06.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_74 /etc/fstab 2020-03-11T22:41:06.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_75 /etc/fstab 2020-03-11T22:41:06.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_76 /etc/fstab 2020-03-11T22:41:06.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_77 /etc/fstab 2020-03-11T22:41:06.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_78 /etc/fstab 2020-03-11T22:41:06.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_79 /etc/fstab 2020-03-11T22:41:06.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_80 /etc/fstab 2020-03-11T22:41:06.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_81 /etc/fstab 2020-03-11T22:41:06.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_82 /etc/fstab 2020-03-11T22:41:06.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_83 /etc/fstab 2020-03-11T22:41:06.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_84 /etc/fstab 2020-03-11T22:41:06.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_85 /etc/fstab 2020-03-11T22:41:06.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:06.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_86 /etc/fstab 2020-03-11T22:41:07.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_87 /etc/fstab 2020-03-11T22:41:07.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_88 /etc/fstab 2020-03-11T22:41:07.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_89 /etc/fstab 2020-03-11T22:41:07.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_90 /etc/fstab 2020-03-11T22:41:07.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_91 /etc/fstab 2020-03-11T22:41:07.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_92 /etc/fstab 2020-03-11T22:41:07.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_93 /etc/fstab 2020-03-11T22:41:07.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_94 /etc/fstab 2020-03-11T22:41:07.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_95 /etc/fstab 2020-03-11T22:41:07.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_96 /etc/fstab 2020-03-11T22:41:07.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_97 /etc/fstab 2020-03-11T22:41:07.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_98 /etc/fstab 2020-03-11T22:41:07.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_99 /etc/fstab 2020-03-11T22:41:07.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:671: TEST_divergent_3: for i in $(seq 1 $testobjects) 2020-03-11T22:41:07.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:673: TEST_divergent_3: rados -p test put existing_100 /etc/fstab 2020-03-11T22:41:08.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:41:08.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:676: TEST_divergent_3: wait_for_clean 2020-03-11T22:41:08.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:41:08.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:41:08.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:41:08.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:41:08.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:41:08.078 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:41:08.078 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:41:08.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:41:08.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:41:08.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:41:08.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:41:08.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:41:08.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:41:08.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:41:08.134 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:41:08.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:41:08.484 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:41:08.484 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:41:08.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:41:08.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:08.484 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:41:08.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:41:08.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:41:08.613 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:08.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:41:08.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705676 2020-03-11T22:41:08.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676' 2020-03-11T22:41:08.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:08.742 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:41:08.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542155 2020-03-11T22:41:08.871 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-60129542155' 2020-03-11T22:41:08.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:08.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:41:08.872 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:08.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:41:08.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:41:08.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:08.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:41:08.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:41:08.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:08.875 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:41:09.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:41:09.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:41:10.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:41:10.237 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:10.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:41:10.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:10.608 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705676 2020-03-11T22:41:10.608 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:10.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:41:10.609 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705676 2020-03-11T22:41:10.610 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:10.610 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705676 2020-03-11T22:41:10.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2020-03-11T22:41:10.611 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:41:10.611 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705676 2020-03-11T22:41:10.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705676 -lt 38654705676 2020-03-11T22:41:10.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:10.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542155 2020-03-11T22:41:10.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:10.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:41:10.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542155 2020-03-11T22:41:10.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:10.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542155 2020-03-11T22:41:10.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2020-03-11T22:41:10.957 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542155 2020-03-11T22:41:10.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:11.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542155 -lt 60129542155 2020-03-11T22:41:11.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:41:11.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:11.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:11.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:41:11.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:11.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:11.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:11.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:11.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:11.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:11.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:12.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:41:12.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:12.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:12.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:12.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:41:12.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:41:12.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:41:12.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: get_pg test existing_1 2020-03-11T22:41:12.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:41:12.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:41:12.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:41:12.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:41:12.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:678: TEST_divergent_3: local pgid=1.0 2020-03-11T22:41:12.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:681: TEST_divergent_3: echo 'blackholing osds 0 2' 2020-03-11T22:41:12.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:682: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:12.874 INFO:tasks.workunit.client.0.smithi155.stdout:blackholing osds 0 2 2020-03-11T22:41:13.202 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:13.202 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:40:50.896158+0000 29'100 29:113 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:41:13.202 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:41:13.203 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:13.203 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:41:13.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:13.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:41:13.209 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:41:13.209 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:41:13.209 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:13.209 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:13.209 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:41:13.210 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.0.asok 2020-03-11T22:41:13.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:13.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:41:13.294 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:41:13.294 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:41:13.294 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:41:13.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:683: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:13.302 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:41:13.302 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:41:13.302 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:41:13.303 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:41:13.303 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:13.303 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:41:13.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.2.asok 2020-03-11T22:41:13.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:41:13.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:685: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:41:13.390 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:41:13.390 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:41:13.390 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:41:13.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:689: TEST_divergent_3: echo 'create new divergent objects' 2020-03-11T22:41:13.396 INFO:tasks.workunit.client.0.smithi155.stdout:create new divergent objects 2020-03-11T22:41:13.397 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: seq 1 2 2020-03-11T22:41:13.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:41:13.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:690: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:41:13.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_1 2020-03-11T22:41:13.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:695: TEST_divergent_3: echo 'writing divergent objects' 2020-03-11T22:41:13.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:692: TEST_divergent_3: rados -p test create newobject_2 2020-03-11T22:41:13.399 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: seq 1 2 2020-03-11T22:41:13.399 INFO:tasks.workunit.client.0.smithi155.stdout:writing divergent objects 2020-03-11T22:41:13.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:41:13.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:696: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_WRITE) 2020-03-11T22:41:13.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_1 /etc/resolv.conf 2020-03-11T22:41:13.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:701: TEST_divergent_3: echo 'remove divergent objects' 2020-03-11T22:41:13.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:698: TEST_divergent_3: rados -p test put existing_2 /etc/resolv.conf 2020-03-11T22:41:13.401 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: seq 1 2 2020-03-11T22:41:13.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:41:13.401 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 1 + 2 2020-03-11T22:41:13.402 INFO:tasks.workunit.client.0.smithi155.stdout:remove divergent objects 2020-03-11T22:41:13.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=3 2020-03-11T22:41:13.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:702: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_REMOVE) 2020-03-11T22:41:13.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_3 2020-03-11T22:41:13.403 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: expr 2 + 2 2020-03-11T22:41:13.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:704: TEST_divergent_3: rmi=4 2020-03-11T22:41:13.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:707: TEST_divergent_3: sleep 10 2020-03-11T22:41:13.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:705: TEST_divergent_3: rados -p test rm existing_4 2020-03-11T22:41:23.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:708: TEST_divergent_3: killall -9 rados 2020-03-11T22:41:23.409 INFO:tasks.workunit.client.0.smithi155.stderr:rados: no process found 2020-03-11T22:41:23.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:711: TEST_divergent_3: echo 'killing all the osds' 2020-03-11T22:41:23.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:712: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:23.410 INFO:tasks.workunit.client.0.smithi155.stdout:killing all the osds 2020-03-11T22:41:23.742 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:23.743 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 active+clean 2020-03-11T22:40:50.896158+0000 29'106 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:41:23.743 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:41:23.743 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:23.744 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:41:23.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:713: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:41:23.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:41:23.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:41:23.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:41:23.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:41:23.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:41:23.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:41:23.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:41:23.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.0 2020-03-11T22:41:24.358 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2020-03-11T22:41:24.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:41:24.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.1 2020-03-11T22:41:24.861 INFO:tasks.workunit.client.0.smithi155.stderr:osd.1 is already down. 2020-03-11T22:41:24.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:714: TEST_divergent_3: for i in $osds 2020-03-11T22:41:24.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:716: TEST_divergent_3: ceph osd down osd.2 2020-03-11T22:41:25.391 INFO:tasks.workunit.client.0.smithi155.stderr:osd.2 is already down. 2020-03-11T22:41:25.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:25.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.0 2020-03-11T22:41:25.893 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already out. 2020-03-11T22:41:25.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:718: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:25.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:720: TEST_divergent_3: ceph osd out osd.2 2020-03-11T22:41:26.397 INFO:tasks.workunit.client.0.smithi155.stderr:osd.2 is already out. 2020-03-11T22:41:26.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:724: TEST_divergent_3: echo 'bringing up non_divergent 0 2' 2020-03-11T22:41:26.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:725: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:41:26.405 INFO:tasks.workunit.client.0.smithi155.stdout:bringing up non_divergent 0 2 2020-03-11T22:41:26.724 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:41:26.724 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 0 0 0 46628 0 0 16 16 stale+active+clean 2020-03-11T22:40:50.896158+0000 29'106 29:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:41:26.724 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:41:26.725 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:41:26.726 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:41:26.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:26.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 0 2020-03-11T22:41:26.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:26.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:26.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:41:26.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:26.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:41:26.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:26.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:26.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:26.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:26.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:41:26.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:41:26.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:26.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:26.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:26.739 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:26.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:26.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:26.740 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:26.740 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:26.740 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:41:26.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:41:26.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:41:26.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:26.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:26.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:26.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:26.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:26.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:26.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:26.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:26.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:41:26.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:41:26.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:26.742 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:41:26.742 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:41:26.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:41:26.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:26.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:26.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:26.763 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:41:26.761+0000 7fbea366dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:26.765 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:41:26.763+0000 7fbea366dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:26.766 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:41:26.764+0000 7fbea366dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:27.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:41:27.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:27.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:41:27.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:27.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:27.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:27.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:27.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:27.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:41:27.084 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:41:27.283 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:41:27.280+0000 7fbea366dec0 -1 Falling back to public interface 2020-03-11T22:41:27.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:41:28.148 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:41:28.145+0000 7fbea366dec0 -1 osd.0 29 log_to_monitors {default=true} 2020-03-11T22:41:28.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:41:28.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:28.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:41:28.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:28.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:41:28.430 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:41:28.777 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up out weight 0 up_from 35 up_thru 0 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/39376,v1:127.0.0.1:6803/39376] [v2:127.0.0.1:6804/39376,v1:127.0.0.1:6805/39376] exists,up 53c048ef-1ae8-4c15-b86c-b15481d0a054 2020-03-11T22:41:28.777 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:41:28.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:28.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:28.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:28.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:726: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:28.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:728: TEST_divergent_3: activate_osd td/divergent-priors 2 2020-03-11T22:41:28.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:41:28.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:41:28.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:41:28.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:41:28.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:41:28.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:41:28.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:41:28.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:41:28.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:41:28.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:41:28.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:41:28.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:41:28.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:41:28.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:41:28.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:41:28.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:41:28.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:41:28.781 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:41:28.781 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:41:28.781 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:41:28.782 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:41:28.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:41:28.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:41:28.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:41:28.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:41:28.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:41:28.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:41:28.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:28.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:41:28.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:41:28.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:41:28.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:41:28.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:41:28.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:41:28.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:41:28.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:41:28.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:41:28.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:41:28.803 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:41:28.800+0000 7fa7cf0afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:28.804 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:41:28.802+0000 7fa7cf0afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:28.806 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:41:28.803+0000 7fa7cf0afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:41:29.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:41:29.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:41:29.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:41:29.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:41:29.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:41:29.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:29.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:41:29.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:41:29.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:29.126 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:41:29.321 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:41:29.319+0000 7fa7cf0afec0 -1 Falling back to public interface 2020-03-11T22:41:29.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:41:30.274 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:41:30.271+0000 7fa7cf0afec0 -1 osd.2 29 log_to_monitors {default=true} 2020-03-11T22:41:30.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:41:30.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:41:30.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:41:30.477 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:41:30.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:41:30.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:41:30.838 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up out weight 0 up_from 36 up_thru 0 down_at 32 last_clean_interval [14,29) [v2:127.0.0.1:6810/39570,v1:127.0.0.1:6811/39570] [v2:127.0.0.1:6812/39570,v1:127.0.0.1:6813/39570] exists,up 8a4738de-77ce-4790-b3b3-87d310c2cbe9 2020-03-11T22:41:30.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:41:30.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:41:30.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:41:30.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:30.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.0 2020-03-11T22:41:31.386 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already in. 2020-03-11T22:41:31.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:730: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:41:31.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:732: TEST_divergent_3: ceph osd in osd.2 2020-03-11T22:41:31.893 INFO:tasks.workunit.client.0.smithi155.stderr:osd.2 is already in. 2020-03-11T22:41:31.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:735: TEST_divergent_3: wait_for_clean 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:41:31.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:41:31.903 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:41:31.903 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:41:31.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:41:31.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:41:31.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:41:31.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:41:31.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:41:31.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:41:31.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:41:31.960 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:41:32.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:41:32.351 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:41:32.352 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:41:32.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:41:32.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:32.352 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:41:32.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855363 2020-03-11T22:41:32.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363' 2020-03-11T22:41:32.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:32.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:41:32.584 INFO:tasks.workunit.client.0.smithi155.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:41:32.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:41:32.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363 1-' 2020-03-11T22:41:32.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:41:32.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:41:32.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822659 2020-03-11T22:41:32.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855363 1- 2-154618822659' 2020-03-11T22:41:32.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:32.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855363 2020-03-11T22:41:32.724 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:32.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:41:32.725 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855363 2020-03-11T22:41:32.725 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:32.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855363 2020-03-11T22:41:32.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855363' 2020-03-11T22:41:32.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:32.726 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 150323855363 2020-03-11T22:41:33.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 150323855363 2020-03-11T22:41:33.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:41:34.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:41:34.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:41:34.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855364 -lt 150323855363 2020-03-11T22:41:34.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:34.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:41:34.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:34.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:41:34.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:41:34.455 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:34.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:41:34.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:41:34.456 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:41:34.457 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 2020-03-11T22:41:34.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705679 -lt 2020-03-11T22:41:34.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705679: unary operator expected 2020-03-11T22:41:34.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:41:34.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822659 2020-03-11T22:41:34.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:41:34.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:41:34.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822659 2020-03-11T22:41:34.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:41:34.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822659 2020-03-11T22:41:34.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822659' 2020-03-11T22:41:34.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:41:34.814 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 154618822659 2020-03-11T22:41:35.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822659 -lt 154618822659 2020-03-11T22:41:35.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:41:35.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:35.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:35.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:41:35.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:35.609 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:35.609 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:35.609 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:35.609 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:35.610 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:35.610 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:35.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:35.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:35.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:35.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:36.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:41:36.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:41:36.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:41:36.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:41:36.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:36.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:41:36.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:36.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:36.500 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:36.500 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:36.500 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:36.500 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:36.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:36.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:36.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:36.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:36.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:36.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:37.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:37.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:37.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:37.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:37.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:37.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:37.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:37.280 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:37.280 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:37.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:37.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:37.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:41:37.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:37.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:41:37.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:37.927 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:37.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:37.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:37.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:37.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:37.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:37.927 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:38.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:38.260 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:38.260 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:38.260 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:38.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:38.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:38.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:38.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:38.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:38.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:38.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:38.723 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:38.723 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:39.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:39.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:39.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:41:39.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:39.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:41:39.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:39.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:39.575 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:39.575 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:39.575 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:39.576 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:39.576 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:39.576 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:39.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:39.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:39.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:39.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:40.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:40.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:40.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:40.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:40.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:40.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:40.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:40.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:40.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:40.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:40.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:40.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:41:40.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:40.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:41:41.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:41.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:41.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:41.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:41.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:41.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:41.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:41.646 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:41.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:41.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:41.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:41.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:42.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:42.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:42.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:42.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:42.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:42.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:42.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:42.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:42.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:42.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:42.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:42.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:41:42.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:42.874 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:41:44.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:44.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:44.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:44.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:44.475 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:44.476 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:44.476 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:44.476 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:44.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:44.819 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:44.820 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:44.820 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:45.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:45.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:45.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:45.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:45.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:45.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:45.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:45.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:45.266 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:45.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:45.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:45.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:41:45.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:45.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:41:48.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:48.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:48.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:48.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:48.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:48.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:48.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:48.909 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:49.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:49.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:49.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:49.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:49.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:49.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:49.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:49.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:49.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:49.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:49.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:49.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:49.698 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:50.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:50.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:50.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:41:50.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:50.137 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:41:56.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:41:56.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:41:56.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:41:56.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:41:56.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:41:56.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:41:56.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:41:56.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:41:56.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:41:56.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:41:56.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:41:56.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:41:57.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:41:57.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:41:57.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:41:57.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:41:57.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:41:57.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:41:57.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:41:57.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:41:57.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:41:57.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:41:57.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:41:57.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:41:57.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:41:57.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:42:05.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:05.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:05.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:05.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:05.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:05.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:05.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:05.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:05.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:05.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:05.438 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:05.438 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:05.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:05.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:05.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:05.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:05.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:05.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:05.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:05.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:05.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:06.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:06.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:06.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:42:06.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:42:06.768 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:42:06.768 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_fingerprint": "38cac1a0-f3fc-4a7e-8be6-955d5897a7d4", 2020-03-11T22:42:06.768 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:42:06.769 INFO:tasks.workunit.client.0.smithi155.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:42:06.769 INFO:tasks.workunit.client.0.smithi155.stdout: "timestamp": "2020-03-11T22:42:06.764988+0000", 2020-03-11T22:42:06.769 INFO:tasks.workunit.client.0.smithi155.stdout: "tag": "", 2020-03-11T22:42:06.769 INFO:tasks.workunit.client.0.smithi155.stdout: "health": { 2020-03-11T22:42:06.769 INFO:tasks.workunit.client.0.smithi155.stdout: "status": "HEALTH_WARN", 2020-03-11T22:42:06.769 INFO:tasks.workunit.client.0.smithi155.stdout: "checks": { 2020-03-11T22:42:06.769 INFO:tasks.workunit.client.0.smithi155.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:42:06.769 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:06.770 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:42:06.770 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:42:06.770 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 17 2020-03-11T22:42:06.770 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.770 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [], 2020-03-11T22:42:06.770 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:42:06.770 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.771 INFO:tasks.workunit.client.0.smithi155.stdout: "OSD_DOWN": { 2020-03-11T22:42:06.771 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:06.771 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:42:06.771 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 osds down", 2020-03-11T22:42:06.771 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:42:06.771 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.771 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:42:06.771 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.772 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "osd.1 (root=default,host=smithi155) is down" 2020-03-11T22:42:06.772 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.772 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.772 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:42:06.772 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.772 INFO:tasks.workunit.client.0.smithi155.stdout: "PG_DEGRADED": { 2020-03-11T22:42:06.772 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:06.773 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:42:06.773 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded", 2020-03-11T22:42:06.773 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 32 2020-03-11T22:42:06.773 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.773 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:42:06.773 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.773 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.773 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.774 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.774 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.774 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.774 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.774 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.2 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.774 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.774 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.774 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.3 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.775 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.775 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.775 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.4 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.775 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.775 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.775 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.5 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.776 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.776 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.776 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.6 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.776 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.776 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.776 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.7 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.776 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.777 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.777 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.8 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.777 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.777 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.777 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.9 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.778 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.778 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.778 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.778 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.778 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.778 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.778 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.778 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.779 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.779 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.779 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.779 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.779 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.779 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.779 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.780 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.780 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.780 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.780 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.780 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.780 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.10 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.780 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.781 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.781 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.11 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.781 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.781 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.781 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.12 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.781 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.781 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.781 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.13 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.782 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.782 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.782 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.14 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.782 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.782 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.782 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.15 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.782 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.783 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.783 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.16 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.783 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.783 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.783 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.17 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.783 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.783 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.784 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.18 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.784 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.784 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.784 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.19 is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.784 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.784 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.784 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1a is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.784 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.785 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.785 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1b is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.785 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.785 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.785 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1c is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.785 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.785 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.786 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1d is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.786 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.786 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.786 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1e is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.786 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.786 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.786 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1f is active+undersized+degraded, acting [0,2]" 2020-03-11T22:42:06.786 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.787 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.787 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:42:06.787 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.787 INFO:tasks.workunit.client.0.smithi155.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:42:06.787 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:06.787 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:42:06.787 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:42:06.787 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:42:06.788 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.788 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:42:06.788 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.788 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:42:06.788 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.788 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.788 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:42:06.789 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.789 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.789 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:42:06.789 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.789 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.789 INFO:tasks.workunit.client.0.smithi155.stdout: "mutes": [] 2020-03-11T22:42:06.789 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.789 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_first_committed": 1, 2020-03-11T22:42:06.790 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_last_committed": 1, 2020-03-11T22:42:06.790 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap": { 2020-03-11T22:42:06.790 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:42:06.790 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:42:06.790 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:40:27.875750Z", 2020-03-11T22:42:06.790 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:40:27.875750Z", 2020-03-11T22:42:06.790 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release": 15, 2020-03-11T22:42:06.791 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:42:06.791 INFO:tasks.workunit.client.0.smithi155.stdout: "features": { 2020-03-11T22:42:06.791 INFO:tasks.workunit.client.0.smithi155.stdout: "persistent": [ 2020-03-11T22:42:06.791 INFO:tasks.workunit.client.0.smithi155.stdout: "kraken", 2020-03-11T22:42:06.791 INFO:tasks.workunit.client.0.smithi155.stdout: "luminous", 2020-03-11T22:42:06.791 INFO:tasks.workunit.client.0.smithi155.stdout: "mimic", 2020-03-11T22:42:06.791 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap-prune", 2020-03-11T22:42:06.791 INFO:tasks.workunit.client.0.smithi155.stdout: "nautilus", 2020-03-11T22:42:06.792 INFO:tasks.workunit.client.0.smithi155.stdout: "octopus" 2020-03-11T22:42:06.792 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.792 INFO:tasks.workunit.client.0.smithi155.stdout: "optional": [] 2020-03-11T22:42:06.792 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.792 INFO:tasks.workunit.client.0.smithi155.stdout: "mons": [ 2020-03-11T22:42:06.792 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.792 INFO:tasks.workunit.client.0.smithi155.stdout: "rank": 0, 2020-03-11T22:42:06.792 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "a", 2020-03-11T22:42:06.793 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:42:06.793 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.793 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.793 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.793 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:42:06.793 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 0 2020-03-11T22:42:06.793 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.794 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.794 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.794 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:42:06.794 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:42:06.794 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 0, 2020-03-11T22:42:06.794 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 0 2020-03-11T22:42:06.794 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.794 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.795 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.795 INFO:tasks.workunit.client.0.smithi155.stdout: "quorum": [ 2020-03-11T22:42:06.795 INFO:tasks.workunit.client.0.smithi155.stdout: 0 2020-03-11T22:42:06.795 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.795 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap": { 2020-03-11T22:42:06.795 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 42, 2020-03-11T22:42:06.795 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:42:06.795 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:40:28.549981+0000", 2020-03-11T22:42:06.796 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:41:34.616654+0000", 2020-03-11T22:42:06.796 INFO:tasks.workunit.client.0.smithi155.stdout: "last_up_change": "2020-03-11T22:41:30.351578+0000", 2020-03-11T22:42:06.796 INFO:tasks.workunit.client.0.smithi155.stdout: "last_in_change": "2020-03-11T22:41:31.709833+0000", 2020-03-11T22:42:06.796 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:42:06.796 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_num": 5800384, 2020-03-11T22:42:06.796 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_set": [ 2020-03-11T22:42:06.796 INFO:tasks.workunit.client.0.smithi155.stdout: "nodown", 2020-03-11T22:42:06.796 INFO:tasks.workunit.client.0.smithi155.stdout: "noin", 2020-03-11T22:42:06.797 INFO:tasks.workunit.client.0.smithi155.stdout: "noout", 2020-03-11T22:42:06.797 INFO:tasks.workunit.client.0.smithi155.stdout: "pglog_hardlimit", 2020-03-11T22:42:06.797 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snapdirs", 2020-03-11T22:42:06.797 INFO:tasks.workunit.client.0.smithi155.stdout: "recovery_deletes", 2020-03-11T22:42:06.797 INFO:tasks.workunit.client.0.smithi155.stdout: "sortbitwise" 2020-03-11T22:42:06.797 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.797 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_version": 7, 2020-03-11T22:42:06.798 INFO:tasks.workunit.client.0.smithi155.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:42:06.798 INFO:tasks.workunit.client.0.smithi155.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:42:06.798 INFO:tasks.workunit.client.0.smithi155.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:42:06.798 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_snapshot": "", 2020-03-11T22:42:06.798 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_max": 1, 2020-03-11T22:42:06.798 INFO:tasks.workunit.client.0.smithi155.stdout: "max_osd": 3, 2020-03-11T22:42:06.798 INFO:tasks.workunit.client.0.smithi155.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:42:06.799 INFO:tasks.workunit.client.0.smithi155.stdout: "min_compat_client": "jewel", 2020-03-11T22:42:06.799 INFO:tasks.workunit.client.0.smithi155.stdout: "require_osd_release": "octopus", 2020-03-11T22:42:06.799 INFO:tasks.workunit.client.0.smithi155.stdout: "pools": [ 2020-03-11T22:42:06.799 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.799 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:42:06.799 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_name": "test", 2020-03-11T22:42:06.799 INFO:tasks.workunit.client.0.smithi155.stdout: "create_time": "2020-03-11T22:40:49.840683+0000", 2020-03-11T22:42:06.799 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": 1, 2020-03-11T22:42:06.800 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_names": "hashpspool", 2020-03-11T22:42:06.800 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2020-03-11T22:42:06.800 INFO:tasks.workunit.client.0.smithi155.stdout: "size": 3, 2020-03-11T22:42:06.800 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 2, 2020-03-11T22:42:06.800 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_rule": 0, 2020-03-11T22:42:06.800 INFO:tasks.workunit.client.0.smithi155.stdout: "object_hash": 2, 2020-03-11T22:42:06.800 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:42:06.800 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num": 32, 2020-03-11T22:42:06.801 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num": 1, 2020-03-11T22:42:06.801 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num_target": 32, 2020-03-11T22:42:06.801 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_target": 32, 2020-03-11T22:42:06.801 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_pending": 32, 2020-03-11T22:42:06.801 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pg_merge_meta": { 2020-03-11T22:42:06.801 INFO:tasks.workunit.client.0.smithi155.stdout: "source_pgid": "0.0", 2020-03-11T22:42:06.801 INFO:tasks.workunit.client.0.smithi155.stdout: "ready_epoch": 0, 2020-03-11T22:42:06.802 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_started": 0, 2020-03-11T22:42:06.802 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_clean": 0, 2020-03-11T22:42:06.802 INFO:tasks.workunit.client.0.smithi155.stdout: "source_version": "0'0", 2020-03-11T22:42:06.802 INFO:tasks.workunit.client.0.smithi155.stdout: "target_version": "0'0" 2020-03-11T22:42:06.802 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.802 INFO:tasks.workunit.client.0.smithi155.stdout: "last_change": "41", 2020-03-11T22:42:06.802 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend": "0", 2020-03-11T22:42:06.803 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:42:06.803 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_preluminous": "41", 2020-03-11T22:42:06.803 INFO:tasks.workunit.client.0.smithi155.stdout: "auid": 0, 2020-03-11T22:42:06.803 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:42:06.803 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_seq": 0, 2020-03-11T22:42:06.803 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_epoch": 0, 2020-03-11T22:42:06.803 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_snaps": [], 2020-03-11T22:42:06.803 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps": "[]", 2020-03-11T22:42:06.804 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_bytes": 0, 2020-03-11T22:42:06.804 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_objects": 0, 2020-03-11T22:42:06.804 INFO:tasks.workunit.client.0.smithi155.stdout: "tiers": [], 2020-03-11T22:42:06.804 INFO:tasks.workunit.client.0.smithi155.stdout: "tier_of": -1, 2020-03-11T22:42:06.804 INFO:tasks.workunit.client.0.smithi155.stdout: "read_tier": -1, 2020-03-11T22:42:06.804 INFO:tasks.workunit.client.0.smithi155.stdout: "write_tier": -1, 2020-03-11T22:42:06.804 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_mode": "none", 2020-03-11T22:42:06.805 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_bytes": 0, 2020-03-11T22:42:06.805 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_objects": 0, 2020-03-11T22:42:06.805 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:42:06.805 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:42:06.805 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:42:06.805 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_flush_age": 0, 2020-03-11T22:42:06.805 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_evict_age": 0, 2020-03-11T22:42:06.805 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profile": "", 2020-03-11T22:42:06.806 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_params": { 2020-03-11T22:42:06.806 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "none" 2020-03-11T22:42:06.806 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.806 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_period": 0, 2020-03-11T22:42:06.806 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_count": 0, 2020-03-11T22:42:06.806 INFO:tasks.workunit.client.0.smithi155.stdout: "use_gmt_hitset": true, 2020-03-11T22:42:06.806 INFO:tasks.workunit.client.0.smithi155.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:42:06.806 INFO:tasks.workunit.client.0.smithi155.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:42:06.807 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:42:06.807 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:42:06.807 INFO:tasks.workunit.client.0.smithi155.stdout: "grade_table": [], 2020-03-11T22:42:06.807 INFO:tasks.workunit.client.0.smithi155.stdout: "stripe_width": 0, 2020-03-11T22:42:06.807 INFO:tasks.workunit.client.0.smithi155.stdout: "expected_num_objects": 0, 2020-03-11T22:42:06.807 INFO:tasks.workunit.client.0.smithi155.stdout: "fast_read": false, 2020-03-11T22:42:06.807 INFO:tasks.workunit.client.0.smithi155.stdout: "options": {}, 2020-03-11T22:42:06.808 INFO:tasks.workunit.client.0.smithi155.stdout: "application_metadata": {} 2020-03-11T22:42:06.808 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.808 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.808 INFO:tasks.workunit.client.0.smithi155.stdout: "osds": [ 2020-03-11T22:42:06.808 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.808 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:42:06.808 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "53c048ef-1ae8-4c15-b86c-b15481d0a054", 2020-03-11T22:42:06.808 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:42:06.809 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:42:06.809 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:42:06.809 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:42:06.809 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 5, 2020-03-11T22:42:06.809 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 29, 2020-03-11T22:42:06.809 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 35, 2020-03-11T22:42:06.809 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 41, 2020-03-11T22:42:06.809 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 30, 2020-03-11T22:42:06.810 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:42:06.810 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:42:06.810 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.810 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.810 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.810 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:42:06.810 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:06.811 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.811 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.811 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:06.811 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:42:06.811 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:06.811 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.811 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.811 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.812 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:42:06.812 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.812 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.812 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.812 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:42:06.812 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:06.812 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.812 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.813 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:06.813 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:42:06.813 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:06.813 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.813 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.813 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.813 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:06.814 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.814 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.814 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.814 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:42:06.814 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:06.814 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.814 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.814 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:06.815 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:42:06.815 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:06.815 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.815 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.815 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.815 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:06.815 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.815 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.816 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.816 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:42:06.816 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:06.816 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.816 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.818 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:06.818 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:42:06.818 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:06.818 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.819 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.819 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.819 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6803/39376", 2020-03-11T22:42:06.819 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6805/39376", 2020-03-11T22:42:06.819 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39376", 2020-03-11T22:42:06.820 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39376", 2020-03-11T22:42:06.820 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:42:06.820 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:42:06.820 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:42:06.820 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.820 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.821 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.821 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:42:06.821 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "ce1120e9-d62c-42de-b0b2-a04b6602228c", 2020-03-11T22:42:06.821 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 0, 2020-03-11T22:42:06.821 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:42:06.821 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:42:06.821 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:42:06.822 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 0, 2020-03-11T22:42:06.822 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 0, 2020-03-11T22:42:06.822 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 9, 2020-03-11T22:42:06.822 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 21, 2020-03-11T22:42:06.822 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 31, 2020-03-11T22:42:06.822 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:42:06.822 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:42:06.822 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.823 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.823 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.823 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:42:06.823 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:06.824 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.824 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.824 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:06.824 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:42:06.824 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:06.825 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.825 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.825 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.825 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:42:06.825 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.825 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.825 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.825 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:42:06.826 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:06.827 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.827 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.827 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:06.827 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:42:06.827 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:06.827 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.827 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.827 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.828 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:06.828 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.828 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.828 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.828 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:42:06.828 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:06.828 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.829 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.829 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:06.829 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:42:06.829 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:06.829 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.829 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.829 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.829 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:06.830 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.830 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.830 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.830 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:42:06.830 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:06.830 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.830 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.831 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:06.831 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:42:06.831 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:06.831 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.831 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.831 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.831 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/34960", 2020-03-11T22:42:06.831 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/34960", 2020-03-11T22:42:06.832 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34960", 2020-03-11T22:42:06.832 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34960", 2020-03-11T22:42:06.832 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:42:06.832 INFO:tasks.workunit.client.0.smithi155.stdout: "exists" 2020-03-11T22:42:06.832 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.832 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.832 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.833 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:42:06.833 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "8a4738de-77ce-4790-b3b3-87d310c2cbe9", 2020-03-11T22:42:06.833 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:42:06.833 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:42:06.833 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:42:06.833 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:42:06.833 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 14, 2020-03-11T22:42:06.834 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 29, 2020-03-11T22:42:06.834 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 36, 2020-03-11T22:42:06.834 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 0, 2020-03-11T22:42:06.834 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 32, 2020-03-11T22:42:06.834 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:42:06.834 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:42:06.834 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.834 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.835 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.835 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:42:06.835 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:06.835 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.835 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.835 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:06.835 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:42:06.836 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:06.836 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.836 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.836 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.836 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:42:06.836 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.836 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.836 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.837 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:42:06.837 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:06.837 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.837 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.837 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:06.837 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:42:06.837 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:06.838 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.838 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.838 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.838 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:06.838 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.838 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.838 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.838 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:42:06.839 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:06.839 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.839 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.839 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:06.839 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:42:06.839 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:06.839 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.840 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.840 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.840 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:06.840 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:06.840 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.840 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:06.840 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:42:06.841 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:06.841 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.841 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.841 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:06.841 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:42:06.841 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:06.841 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.842 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.842 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.842 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/39570", 2020-03-11T22:42:06.842 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/39570", 2020-03-11T22:42:06.842 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39570", 2020-03-11T22:42:06.842 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39570", 2020-03-11T22:42:06.842 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:42:06.842 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:42:06.843 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:42:06.843 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.843 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.843 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.843 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_xinfo": [ 2020-03-11T22:42:06.843 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.843 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:42:06.844 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:41:24.154203+0000", 2020-03-11T22:42:06.844 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:42:06.844 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:42:06.845 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:42:06.845 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:42:06.845 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:35.219964+0000", 2020-03-11T22:42:06.845 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:42:06.845 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.846 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.846 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:42:06.846 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:41:24.661922+0000", 2020-03-11T22:42:06.846 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:42:06.846 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:42:06.846 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:42:06.847 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:42:06.847 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:41.515052+0000", 2020-03-11T22:42:06.847 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:42:06.848 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.848 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.848 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:42:06.848 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:41:25.163731+0000", 2020-03-11T22:42:06.848 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:42:06.848 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:42:06.848 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:42:06.848 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:42:06.849 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:47.907713+0000", 2020-03-11T22:42:06.849 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:42:06.849 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.849 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.849 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap": [], 2020-03-11T22:42:06.849 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap_items": [], 2020-03-11T22:42:06.849 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_temp": [], 2020-03-11T22:42:06.850 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_temp": [], 2020-03-11T22:42:06.850 INFO:tasks.workunit.client.0.smithi155.stdout: "blacklist": {}, 2020-03-11T22:42:06.850 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profiles": { 2020-03-11T22:42:06.850 INFO:tasks.workunit.client.0.smithi155.stdout: "default": { 2020-03-11T22:42:06.850 INFO:tasks.workunit.client.0.smithi155.stdout: "k": "2", 2020-03-11T22:42:06.850 INFO:tasks.workunit.client.0.smithi155.stdout: "m": "2", 2020-03-11T22:42:06.850 INFO:tasks.workunit.client.0.smithi155.stdout: "plugin": "jerasure", 2020-03-11T22:42:06.851 INFO:tasks.workunit.client.0.smithi155.stdout: "technique": "reed_sol_van" 2020-03-11T22:42:06.851 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.851 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.851 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps_queue": [], 2020-03-11T22:42:06.851 INFO:tasks.workunit.client.0.smithi155.stdout: "new_removed_snaps": [], 2020-03-11T22:42:06.851 INFO:tasks.workunit.client.0.smithi155.stdout: "new_purged_snaps": [], 2020-03-11T22:42:06.851 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_node_flags": {}, 2020-03-11T22:42:06.851 INFO:tasks.workunit.client.0.smithi155.stdout: "device_class_flags": {} 2020-03-11T22:42:06.852 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.852 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_metadata": [ 2020-03-11T22:42:06.852 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.852 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:42:06.852 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:42:06.852 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6804/39376,v1:127.0.0.1:6805/39376]", 2020-03-11T22:42:06.852 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:42:06.852 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:42:06.853 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:06.853 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:06.853 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:06.853 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:06.853 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:06.853 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:06.854 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:06.854 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:42:06.854 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:06.854 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:06.854 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:06.854 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:06.854 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:42:06.854 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:06.855 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:06.855 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:06.855 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:42:06.855 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:42:06.855 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:06.855 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:42:06.855 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:42:06.856 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:06.856 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:42:06.856 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6802/39376,v1:127.0.0.1:6803/39376]", 2020-03-11T22:42:06.856 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:42:06.856 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39376,v1:127.0.0.1:6809/39376]", 2020-03-11T22:42:06.856 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39376,v1:127.0.0.1:6807/39376]", 2020-03-11T22:42:06.856 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:42:06.856 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:42:06.857 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:06.857 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:06.857 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:06.857 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:42:06.857 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:06.857 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:06.857 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:42:06.858 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:42:06.858 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:06.858 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:42:06.858 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.858 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.858 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:42:06.858 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:42:06.858 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/34960,v1:127.0.0.1:6813/34960]", 2020-03-11T22:42:06.859 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:42:06.859 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:42:06.859 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:06.859 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:06.859 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:06.859 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:06.859 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:06.860 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:06.860 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:06.860 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:42:06.860 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:06.860 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:06.860 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:06.860 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:06.860 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:42:06.861 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:06.861 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:06.861 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:06.861 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:42:06.861 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:42:06.861 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:06.861 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:42:06.862 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:42:06.862 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:06.862 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:42:06.862 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/34960,v1:127.0.0.1:6811/34960]", 2020-03-11T22:42:06.862 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:42:06.862 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34960,v1:127.0.0.1:6817/34960]", 2020-03-11T22:42:06.862 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34960,v1:127.0.0.1:6815/34960]", 2020-03-11T22:42:06.862 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:42:06.863 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:42:06.863 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:06.863 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:06.863 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:06.863 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:42:06.863 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:06.863 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:06.864 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:42:06.864 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:42:06.864 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:06.864 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:42:06.864 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.864 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.864 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:42:06.865 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:42:06.865 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/39570,v1:127.0.0.1:6813/39570]", 2020-03-11T22:42:06.865 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:42:06.865 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:42:06.865 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:06.865 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:06.865 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:06.865 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:06.866 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:06.866 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:06.866 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:06.866 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:42:06.866 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:06.866 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:06.866 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:06.867 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:06.867 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:42:06.867 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:06.867 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:06.867 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:06.867 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:42:06.867 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:42:06.867 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:06.868 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:42:06.868 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:42:06.868 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:06.868 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:42:06.868 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/39570,v1:127.0.0.1:6811/39570]", 2020-03-11T22:42:06.868 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:42:06.868 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39570,v1:127.0.0.1:6817/39570]", 2020-03-11T22:42:06.869 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39570,v1:127.0.0.1:6815/39570]", 2020-03-11T22:42:06.869 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:42:06.869 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:42:06.869 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:06.869 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:06.869 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:06.869 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:42:06.869 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:06.870 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:06.870 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:42:06.870 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:42:06.870 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:06.870 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:42:06.870 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.870 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.871 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_first_committed": 1, 2020-03-11T22:42:06.871 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_last_committed": 42, 2020-03-11T22:42:06.871 INFO:tasks.workunit.client.0.smithi155.stdout: "crushmap": { 2020-03-11T22:42:06.871 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": [ 2020-03-11T22:42:06.871 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.871 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:42:06.871 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.0", 2020-03-11T22:42:06.871 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:42:06.872 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.872 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.872 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:42:06.872 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.1", 2020-03-11T22:42:06.872 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:42:06.872 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.872 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.872 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:42:06.873 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.2", 2020-03-11T22:42:06.873 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:42:06.873 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.873 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.873 INFO:tasks.workunit.client.0.smithi155.stdout: "types": [ 2020-03-11T22:42:06.873 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.873 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 0, 2020-03-11T22:42:06.874 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd" 2020-03-11T22:42:06.874 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.874 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.874 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:42:06.874 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "host" 2020-03-11T22:42:06.874 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.874 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.874 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 2, 2020-03-11T22:42:06.875 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "chassis" 2020-03-11T22:42:06.875 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.875 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.875 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 3, 2020-03-11T22:42:06.875 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "rack" 2020-03-11T22:42:06.875 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.875 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.876 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 4, 2020-03-11T22:42:06.876 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "row" 2020-03-11T22:42:06.876 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.876 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.876 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 5, 2020-03-11T22:42:06.876 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pdu" 2020-03-11T22:42:06.876 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.876 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.877 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 6, 2020-03-11T22:42:06.877 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pod" 2020-03-11T22:42:06.877 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.877 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.877 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 7, 2020-03-11T22:42:06.877 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "room" 2020-03-11T22:42:06.877 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.878 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.878 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 8, 2020-03-11T22:42:06.878 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "datacenter" 2020-03-11T22:42:06.878 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.878 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.878 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 9, 2020-03-11T22:42:06.878 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "zone" 2020-03-11T22:42:06.878 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.879 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.879 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 10, 2020-03-11T22:42:06.879 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "region" 2020-03-11T22:42:06.879 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.879 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.879 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:42:06.879 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "root" 2020-03-11T22:42:06.879 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.880 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.880 INFO:tasks.workunit.client.0.smithi155.stdout: "buckets": [ 2020-03-11T22:42:06.880 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.880 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -1, 2020-03-11T22:42:06.880 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default", 2020-03-11T22:42:06.880 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:42:06.880 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:42:06.881 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:42:06.881 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:42:06.881 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:42:06.881 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:42:06.881 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.881 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:42:06.881 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:42:06.881 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:42:06.882 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.882 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.882 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.882 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.882 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -2, 2020-03-11T22:42:06.882 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default~hdd", 2020-03-11T22:42:06.882 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:42:06.883 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:42:06.883 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:42:06.883 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:42:06.883 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:42:06.883 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:42:06.883 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.883 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:42:06.883 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:42:06.884 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:42:06.884 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.884 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.884 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.884 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.884 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:42:06.884 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155", 2020-03-11T22:42:06.884 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:42:06.885 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:42:06.885 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:42:06.885 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:42:06.885 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:42:06.885 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:42:06.885 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.885 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:42:06.886 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:42:06.886 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:42:06.886 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.886 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.886 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:42:06.886 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:42:06.886 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:42:06.886 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.887 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.887 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:42:06.887 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:42:06.887 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:42:06.887 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.887 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.887 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.887 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.888 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:42:06.888 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155~hdd", 2020-03-11T22:42:06.888 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:42:06.888 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:42:06.888 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:42:06.888 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:42:06.888 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:42:06.889 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:42:06.889 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.889 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:42:06.889 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:42:06.889 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:42:06.889 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.889 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.889 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:42:06.890 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:42:06.890 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:42:06.890 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.890 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.890 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:42:06.890 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:42:06.890 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:42:06.891 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.891 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.891 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.891 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.891 INFO:tasks.workunit.client.0.smithi155.stdout: "rules": [ 2020-03-11T22:42:06.891 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.891 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_id": 0, 2020-03-11T22:42:06.892 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_name": "replicated_rule", 2020-03-11T22:42:06.892 INFO:tasks.workunit.client.0.smithi155.stdout: "ruleset": 0, 2020-03-11T22:42:06.892 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2020-03-11T22:42:06.892 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 1, 2020-03-11T22:42:06.892 INFO:tasks.workunit.client.0.smithi155.stdout: "max_size": 10, 2020-03-11T22:42:06.892 INFO:tasks.workunit.client.0.smithi155.stdout: "steps": [ 2020-03-11T22:42:06.892 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.893 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "take", 2020-03-11T22:42:06.893 INFO:tasks.workunit.client.0.smithi155.stdout: "item": -1, 2020-03-11T22:42:06.893 INFO:tasks.workunit.client.0.smithi155.stdout: "item_name": "default" 2020-03-11T22:42:06.893 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.893 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.893 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "choose_firstn", 2020-03-11T22:42:06.893 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 0, 2020-03-11T22:42:06.893 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "osd" 2020-03-11T22:42:06.894 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.894 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.894 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "emit" 2020-03-11T22:42:06.894 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.894 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:06.894 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.894 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.895 INFO:tasks.workunit.client.0.smithi155.stdout: "tunables": { 2020-03-11T22:42:06.895 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_tries": 0, 2020-03-11T22:42:06.895 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:42:06.895 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_total_tries": 50, 2020-03-11T22:42:06.895 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:42:06.895 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:42:06.895 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_stable": 1, 2020-03-11T22:42:06.895 INFO:tasks.workunit.client.0.smithi155.stdout: "straw_calc_version": 1, 2020-03-11T22:42:06.896 INFO:tasks.workunit.client.0.smithi155.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:42:06.896 INFO:tasks.workunit.client.0.smithi155.stdout: "profile": "jewel", 2020-03-11T22:42:06.896 INFO:tasks.workunit.client.0.smithi155.stdout: "optimal_tunables": 1, 2020-03-11T22:42:06.896 INFO:tasks.workunit.client.0.smithi155.stdout: "legacy_tunables": 0, 2020-03-11T22:42:06.896 INFO:tasks.workunit.client.0.smithi155.stdout: "minimum_required_version": "jewel", 2020-03-11T22:42:06.896 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables": 1, 2020-03-11T22:42:06.896 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables2": 1, 2020-03-11T22:42:06.897 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v2_rules": 0, 2020-03-11T22:42:06.897 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables3": 1, 2020-03-11T22:42:06.897 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v3_rules": 0, 2020-03-11T22:42:06.897 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v4_buckets": 1, 2020-03-11T22:42:06.897 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables5": 1, 2020-03-11T22:42:06.897 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v5_rules": 0 2020-03-11T22:42:06.897 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.897 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_args": {} 2020-03-11T22:42:06.898 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.898 INFO:tasks.workunit.client.0.smithi155.stdout: "fsmap": { 2020-03-11T22:42:06.898 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:42:06.898 INFO:tasks.workunit.client.0.smithi155.stdout: "default_fscid": -1, 2020-03-11T22:42:06.898 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": { 2020-03-11T22:42:06.898 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": {}, 2020-03-11T22:42:06.898 INFO:tasks.workunit.client.0.smithi155.stdout: "ro_compat": {}, 2020-03-11T22:42:06.899 INFO:tasks.workunit.client.0.smithi155.stdout: "incompat": { 2020-03-11T22:42:06.899 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_1": "base v0.20", 2020-03-11T22:42:06.899 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:42:06.899 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:42:06.899 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:42:06.899 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:42:06.899 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:42:06.899 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_8": "no anchor table", 2020-03-11T22:42:06.900 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_9": "file layout v2", 2020-03-11T22:42:06.900 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:42:06.900 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.900 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.900 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_flags": { 2020-03-11T22:42:06.900 INFO:tasks.workunit.client.0.smithi155.stdout: "enable_multiple": false, 2020-03-11T22:42:06.900 INFO:tasks.workunit.client.0.smithi155.stdout: "ever_enabled_multiple": false 2020-03-11T22:42:06.901 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.901 INFO:tasks.workunit.client.0.smithi155.stdout: "standbys": [], 2020-03-11T22:42:06.901 INFO:tasks.workunit.client.0.smithi155.stdout: "filesystems": [] 2020-03-11T22:42:06.901 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.901 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:42:06.901 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:42:06.901 INFO:tasks.workunit.client.0.smithi155.stdout: "auth": { 2020-03-11T22:42:06.901 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:42:06.902 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 5, 2020-03-11T22:42:06.902 INFO:tasks.workunit.client.0.smithi155.stdout: "num_secrets": 3 2020-03-11T22:42:06.902 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.902 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 32, 2020-03-11T22:42:06.902 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_active": 32, 2020-03-11T22:42:06.902 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_unknown": 0, 2020-03-11T22:42:06.902 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osd": 3, 2020-03-11T22:42:06.902 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_sum": { 2020-03-11T22:42:06.903 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:42:06.903 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 48300, 2020-03-11T22:42:06.903 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 100, 2020-03-11T22:42:06.903 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2020-03-11T22:42:06.903 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 300, 2020-03-11T22:42:06.903 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:42:06.903 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:42:06.904 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 100, 2020-03-11T22:42:06.904 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:42:06.904 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:42:06.904 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 100, 2020-03-11T22:42:06.904 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:42:06.904 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:42:06.904 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:42:06.904 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 100, 2020-03-11T22:42:06.905 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 100, 2020-03-11T22:42:06.905 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:42:06.905 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:42:06.905 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:42:06.905 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:42:06.905 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:42:06.905 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:42:06.906 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:42:06.906 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:42:06.906 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:42:06.906 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:42:06.906 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:42:06.906 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:42:06.906 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:42:06.906 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:42:06.907 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:42:06.907 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:42:06.907 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:42:06.907 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:42:06.907 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:42:06.907 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:42:06.907 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:42:06.908 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:42:06.908 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:42:06.908 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:42:06.908 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:42:06.908 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.908 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:42:06.908 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:42:06.908 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:42:06.909 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:42:06.909 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 0, 2020-03-11T22:42:06.909 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 0, 2020-03-11T22:42:06.909 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:42:06.909 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:06.909 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:42:06.909 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:42:06.910 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:42:06.910 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.910 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 512, 2020-03-11T22:42:06.910 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 512, 2020-03-11T22:42:06.910 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 64, 2020-03-11T22:42:06.910 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 64, 2020-03-11T22:42:06.910 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 0 2020-03-11T22:42:06.910 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.911 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum": { 2020-03-11T22:42:06.911 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:42:06.911 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:42:06.911 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 65, 2020-03-11T22:42:06.911 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:42:06.911 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:42:06.911 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:42:06.912 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:42:06.912 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3179968, 2020-03-11T22:42:06.912 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 33920, 2020-03-11T22:42:06.912 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 14, 2020-03-11T22:42:06.912 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145713, 2020-03-11T22:42:06.912 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311392832, 2020-03-11T22:42:06.912 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:42:06.912 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:42:06.913 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318866259968, 2020-03-11T22:42:06.913 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:42:06.913 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 34734080, 2020-03-11T22:42:06.913 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 427731, 2020-03-11T22:42:06.913 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:42:06.913 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:06.913 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:42:06.913 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 14646, 2020-03-11T22:42:06.914 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221210826 2020-03-11T22:42:06.914 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.914 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:42:06.914 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:42:06.914 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:42:06.914 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2020-03-11T22:42:06.914 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:42:06.915 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:42:06.915 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:42:06.915 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.915 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:42:06.915 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:42:06.915 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:42:06.915 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:42:06.915 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:42:06.916 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.916 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:42:06.916 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:42:06.916 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.916 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum_by_class": { 2020-03-11T22:42:06.916 INFO:tasks.workunit.client.0.smithi155.stdout: "hdd": { 2020-03-11T22:42:06.916 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:42:06.916 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:42:06.917 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 65, 2020-03-11T22:42:06.917 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:42:06.917 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:42:06.917 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:42:06.917 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:42:06.917 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3179968, 2020-03-11T22:42:06.917 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 33920, 2020-03-11T22:42:06.918 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 14, 2020-03-11T22:42:06.918 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145713, 2020-03-11T22:42:06.918 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311392832, 2020-03-11T22:42:06.918 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:42:06.918 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:42:06.918 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318866259968, 2020-03-11T22:42:06.918 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:42:06.919 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 34734080, 2020-03-11T22:42:06.919 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 427731, 2020-03-11T22:42:06.919 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:42:06.919 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:06.919 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:42:06.919 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 14646, 2020-03-11T22:42:06.919 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221210826 2020-03-11T22:42:06.920 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.920 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:42:06.920 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:42:06.920 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:42:06.920 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2020-03-11T22:42:06.920 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:42:06.920 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:42:06.921 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:42:06.921 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.921 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:42:06.921 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:42:06.921 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:42:06.921 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:42:06.921 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:42:06.921 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.922 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:42:06.922 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:42:06.922 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.922 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.922 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_stats": [ 2020-03-11T22:42:06.922 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.922 INFO:tasks.workunit.client.0.smithi155.stdout: "poolid": 1, 2020-03-11T22:42:06.923 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 32, 2020-03-11T22:42:06.923 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:42:06.923 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 48300, 2020-03-11T22:42:06.923 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 100, 2020-03-11T22:42:06.923 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2020-03-11T22:42:06.923 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 300, 2020-03-11T22:42:06.923 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:42:06.923 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:42:06.924 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 100, 2020-03-11T22:42:06.924 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:42:06.924 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:42:06.924 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 100, 2020-03-11T22:42:06.924 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:42:06.924 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:42:06.924 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:42:06.924 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 100, 2020-03-11T22:42:06.925 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 100, 2020-03-11T22:42:06.925 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:42:06.925 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:42:06.925 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:42:06.925 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:42:06.925 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:42:06.925 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:42:06.926 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:42:06.926 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:42:06.926 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:42:06.926 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:42:06.926 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:42:06.926 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:42:06.926 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:42:06.926 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:42:06.927 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:42:06.927 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:42:06.927 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:42:06.927 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:42:06.927 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:42:06.927 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:42:06.927 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:42:06.928 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:42:06.928 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:42:06.928 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:42:06.928 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:42:06.928 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.928 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:42:06.928 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:42:06.928 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:42:06.929 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:42:06.929 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 19529728, 2020-03-11T22:42:06.929 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 143228, 2020-03-11T22:42:06.929 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:42:06.929 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:06.929 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:42:06.929 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:42:06.930 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:42:06.930 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.930 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 512, 2020-03-11T22:42:06.930 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 512, 2020-03-11T22:42:06.930 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 64, 2020-03-11T22:42:06.930 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 64, 2020-03-11T22:42:06.930 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 3 2020-03-11T22:42:06.930 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.931 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.931 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_stats": [ 2020-03-11T22:42:06.931 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.931 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:42:06.931 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 150323855370 2020-03-11T22:42:06.931 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.931 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.931 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:42:06.932 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 38654705679 2020-03-11T22:42:06.932 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.932 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.932 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:42:06.932 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 154618822666 2020-03-11T22:42:06.932 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.932 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.932 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_state": [ 2020-03-11T22:42:06.933 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.933 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "active+undersized+degraded", 2020-03-11T22:42:06.933 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 32 2020-03-11T22:42:06.933 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.933 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.933 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_osd": [ 2020-03-11T22:42:06.933 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.934 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:42:06.934 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:42:06.934 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 32, 2020-03-11T22:42:06.934 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:06.934 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.934 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.934 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:42:06.934 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:42:06.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 0, 2020-03-11T22:42:06.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:06.935 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.935 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.935 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:42:06.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 32, 2020-03-11T22:42:06.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 32, 2020-03-11T22:42:06.935 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:06.936 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.936 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.936 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": [ 2020-03-11T22:42:06.936 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:06.936 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:42:06.936 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": {} 2020-03-11T22:42:06.936 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.937 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:06.937 INFO:tasks.workunit.client.0.smithi155.stdout: "servicemap": { 2020-03-11T22:42:06.937 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:42:06.937 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:40:28.550306+0000", 2020-03-11T22:42:06.937 INFO:tasks.workunit.client.0.smithi155.stdout: "services": {} 2020-03-11T22:42:06.937 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:06.937 INFO:tasks.workunit.client.0.smithi155.stdout: "paxos": { 2020-03-11T22:42:06.937 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:42:06.938 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 308, 2020-03-11T22:42:06.938 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pn": 0, 2020-03-11T22:42:06.938 INFO:tasks.workunit.client.0.smithi155.stdout: "accepted_pn": 0 2020-03-11T22:42:06.938 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:06.938 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:42:06.939 INFO:tasks.workunit.client.0.smithi155.stderr:report 1656152222 2020-03-11T22:42:06.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:42:06.939 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: expr 2 + 2 2020-03-11T22:42:06.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:738: TEST_divergent_3: objname=existing_4 2020-03-11T22:42:06.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:739: TEST_divergent_3: echo 'writing non-divergent object existing_4' 2020-03-11T22:42:06.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:740: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:06.940 INFO:tasks.workunit.client.0.smithi155.stdout:writing non-divergent object existing_4 2020-03-11T22:42:07.102 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:07.102 INFO:tasks.workunit.client.0.smithi155.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.929055+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.102 INFO:tasks.workunit.client.0.smithi155.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928394+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.102 INFO:tasks.workunit.client.0.smithi155.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928671+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.103 INFO:tasks.workunit.client.0.smithi155.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928362+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.103 INFO:tasks.workunit.client.0.smithi155.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928831+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.103 INFO:tasks.workunit.client.0.smithi155.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926198+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.103 INFO:tasks.workunit.client.0.smithi155.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926228+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.103 INFO:tasks.workunit.client.0.smithi155.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928059+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.103 INFO:tasks.workunit.client.0.smithi155.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927991+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.103 INFO:tasks.workunit.client.0.smithi155.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928167+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.104 INFO:tasks.workunit.client.0.smithi155.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927262+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.104 INFO:tasks.workunit.client.0.smithi155.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926848+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.104 INFO:tasks.workunit.client.0.smithi155.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927687+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.104 INFO:tasks.workunit.client.0.smithi155.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927622+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.104 INFO:tasks.workunit.client.0.smithi155.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927785+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.104 INFO:tasks.workunit.client.0.smithi155.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927684+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.104 INFO:tasks.workunit.client.0.smithi155.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927453+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.104 INFO:tasks.workunit.client.0.smithi155.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927314+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.105 INFO:tasks.workunit.client.0.smithi155.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926849+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.105 INFO:tasks.workunit.client.0.smithi155.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927401+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.105 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:48.076964+0000 29'100 42:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.105 INFO:tasks.workunit.client.0.smithi155.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928094+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.105 INFO:tasks.workunit.client.0.smithi155.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928720+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.105 INFO:tasks.workunit.client.0.smithi155.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926387+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.105 INFO:tasks.workunit.client.0.smithi155.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926235+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.106 INFO:tasks.workunit.client.0.smithi155.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926193+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.106 INFO:tasks.workunit.client.0.smithi155.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.929022+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.106 INFO:tasks.workunit.client.0.smithi155.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926841+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.106 INFO:tasks.workunit.client.0.smithi155.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926989+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.106 INFO:tasks.workunit.client.0.smithi155.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927241+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.106 INFO:tasks.workunit.client.0.smithi155.stdout:1.e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926847+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.106 INFO:tasks.workunit.client.0.smithi155.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927819+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:07.107 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:42:07.107 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:07.107 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:42:07.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:741: TEST_divergent_3: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:42:07.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:42:07.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:743: TEST_divergent_3: wait_for_clean 2020-03-11T22:42:07.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:42:07.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:42:07.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:42:07.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:42:07.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:42:07.165 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:42:07.165 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:42:07.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:42:07.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:42:07.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:42:07.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:42:07.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:42:07.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:42:07.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:42:07.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:42:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:42:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:42:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:42:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:42:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:07.566 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:42:07.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855372 2020-03-11T22:42:07.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855372' 2020-03-11T22:42:07.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:07.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:42:07.804 INFO:tasks.workunit.client.0.smithi155.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:42:07.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:42:07.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855372 1-' 2020-03-11T22:42:07.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:42:07.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:42:07.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=154618822668 2020-03-11T22:42:07.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-150323855372 1- 2-154618822668' 2020-03-11T22:42:07.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:07.963 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-150323855372 2020-03-11T22:42:07.963 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:07.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:42:07.965 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-150323855372 2020-03-11T22:42:07.965 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:07.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855372 2020-03-11T22:42:07.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 150323855372' 2020-03-11T22:42:07.967 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 150323855372 2020-03-11T22:42:07.967 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:08.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855370 -lt 150323855372 2020-03-11T22:42:08.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:42:09.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:42:09.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:42:09.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855373 -lt 150323855372 2020-03-11T22:42:09.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:09.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:42:09.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:09.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:42:09.646 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:42:09.646 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:09.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:42:09.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:42:09.647 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 2020-03-11T22:42:09.648 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:42:10.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705679 -lt 2020-03-11T22:42:10.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705679: unary operator expected 2020-03-11T22:42:10.021 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:42:10.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-154618822668 2020-03-11T22:42:10.022 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:42:10.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:42:10.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-154618822668 2020-03-11T22:42:10.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:42:10.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=154618822668 2020-03-11T22:42:10.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 154618822668' 2020-03-11T22:42:10.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:42:10.026 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 154618822668 2020-03-11T22:42:10.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 154618822668 -lt 154618822668 2020-03-11T22:42:10.374 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:42:10.374 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:10.374 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:10.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 32 == 0 2020-03-11T22:42:10.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:10.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:10.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:10.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:10.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:10.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:10.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:11.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:11.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:11.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:11.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:11.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:11.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:42:11.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:42:11.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:42:11.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:11.610 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:42:11.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:11.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:11.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:11.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:11.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:11.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:11.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:11.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:12.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:12.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:12.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:12.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:12.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:12.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:12.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:12.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:12.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:12.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:12.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:12.496 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:12.496 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:12.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:12.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:12.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:42:12.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:12.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:42:13.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:13.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:13.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:13.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:13.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:13.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:13.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:13.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:13.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:13.481 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:13.481 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:13.481 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:13.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:13.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:13.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:13.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:13.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:13.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:13.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:13.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:13.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:14.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:14.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:14.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:42:14.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:14.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:42:14.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:14.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:14.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:14.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:14.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:14.773 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:14.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:14.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:15.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:15.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:15.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:15.115 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:15.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:15.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:15.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:15.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:15.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:15.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:15.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:15.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:15.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:16.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:16.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:16.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:42:16.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:16.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:42:16.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:16.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:16.802 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:16.802 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:16.802 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:16.802 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:16.803 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:16.803 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:17.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:17.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:17.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:17.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:17.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:17.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:17.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:17.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:17.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:17.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:17.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:17.590 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:17.590 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:18.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:18.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:18.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:42:18.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:18.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:42:19.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:19.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:19.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:19.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:19.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:19.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:19.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:19.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:19.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:19.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:19.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:19.975 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:20.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:20.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:20.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:20.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:20.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:20.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:20.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:20.418 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:20.418 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:20.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:20.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:20.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:42:20.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:20.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:42:24.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:24.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:24.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:24.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:24.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:24.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:24.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:24.080 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:24.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:24.426 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:24.426 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:24.427 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:24.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:24.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:24.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:24.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:24.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:24.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:24.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:24.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:24.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:25.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:25.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:25.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:42:25.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:25.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:42:31.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:31.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:31.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:31.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:31.716 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:31.716 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:31.716 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:31.716 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:32.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:32.062 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:32.062 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:32.062 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:32.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:32.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:32.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:32.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:32.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:32.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:32.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:32.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:32.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:32.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:32.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:32.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:42:32.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:42:32.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:42:40.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:42:40.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:42:40.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:42:40.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:42:40.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:42:40.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:42:40.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:42:40.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:42:40.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:42:40.634 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:42:40.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:42:40.635 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:42:41.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 32 2020-03-11T22:42:41.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:42:41.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:42:41.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:42:41.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:42:41.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:42:41.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:42:41.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:42:41.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:42:41.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:42:41.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:42:41.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:42:41.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:42:41.948 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:42:41.948 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_fingerprint": "38cac1a0-f3fc-4a7e-8be6-955d5897a7d4", 2020-03-11T22:42:41.949 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:42:41.949 INFO:tasks.workunit.client.0.smithi155.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:42:41.949 INFO:tasks.workunit.client.0.smithi155.stdout: "timestamp": "2020-03-11T22:42:41.945151+0000", 2020-03-11T22:42:41.949 INFO:tasks.workunit.client.0.smithi155.stdout: "tag": "", 2020-03-11T22:42:41.949 INFO:tasks.workunit.client.0.smithi155.stdout: "health": { 2020-03-11T22:42:41.949 INFO:tasks.workunit.client.0.smithi155.stdout: "status": "HEALTH_WARN", 2020-03-11T22:42:41.949 INFO:tasks.workunit.client.0.smithi155.stdout: "checks": { 2020-03-11T22:42:41.950 INFO:tasks.workunit.client.0.smithi155.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:42:41.950 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:41.950 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:42:41.950 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:42:41.950 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 17 2020-03-11T22:42:41.950 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.950 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [], 2020-03-11T22:42:41.951 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:42:41.951 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.951 INFO:tasks.workunit.client.0.smithi155.stdout: "OSD_DOWN": { 2020-03-11T22:42:41.951 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:41.951 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:42:41.951 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 osds down", 2020-03-11T22:42:41.951 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:42:41.951 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.952 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:42:41.952 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.952 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "osd.1 (root=default,host=smithi155) is down" 2020-03-11T22:42:41.952 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:41.952 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:41.952 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:42:41.952 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.953 INFO:tasks.workunit.client.0.smithi155.stdout: "PG_DEGRADED": { 2020-03-11T22:42:41.953 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:41.953 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:42:41.953 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "Degraded data redundancy: 100/300 objects degraded (33.333%), 32 pgs degraded, 32 pgs undersized", 2020-03-11T22:42:41.953 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 64 2020-03-11T22:42:41.953 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.953 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:42:41.954 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.954 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.0 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.954 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.954 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.954 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.954 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.954 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.954 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.2 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.955 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.955 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.955 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.3 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.955 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.955 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.955 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.4 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.955 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.956 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.956 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.5 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.956 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.959 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.959 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.6 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.959 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.960 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.960 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.7 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.960 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.960 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.960 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.8 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.960 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.960 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.961 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.9 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.961 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.961 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.961 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.a is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.961 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.961 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.961 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.b is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.962 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.962 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.962 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.c is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.962 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.962 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.962 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.d is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.962 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.963 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.963 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.e is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.963 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.963 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.963 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.f is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.963 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.963 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.964 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.10 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.964 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.964 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.964 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.11 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.964 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.964 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.965 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.12 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.965 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.965 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.965 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.13 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.965 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.965 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.966 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.14 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.966 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.966 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.966 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.15 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.966 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.966 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.966 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.16 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.967 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.967 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.967 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.17 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.967 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.967 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.967 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.18 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.967 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.967 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.968 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.19 is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.968 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.968 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.968 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1a is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.968 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.968 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.968 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1b is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.969 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.969 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.969 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1c is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.969 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.969 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.969 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1d is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.970 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.970 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.970 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1e is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.970 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.970 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.970 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.1f is stuck undersized for 65s, current state active+undersized+degraded, last acting [0,2]" 2020-03-11T22:42:41.970 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:41.971 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:41.971 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:42:41.971 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.971 INFO:tasks.workunit.client.0.smithi155.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:42:41.971 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:42:41.971 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:42:41.971 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:42:41.971 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:42:41.972 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.972 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:42:41.972 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.972 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:42:41.972 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.972 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.972 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:42:41.973 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:41.973 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:41.973 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:42:41.973 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:41.973 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.973 INFO:tasks.workunit.client.0.smithi155.stdout: "mutes": [] 2020-03-11T22:42:41.973 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.973 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_first_committed": 1, 2020-03-11T22:42:41.974 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_last_committed": 1, 2020-03-11T22:42:41.974 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap": { 2020-03-11T22:42:41.974 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:42:41.974 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:42:41.974 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:40:27.875750Z", 2020-03-11T22:42:41.974 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:40:27.875750Z", 2020-03-11T22:42:41.974 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release": 15, 2020-03-11T22:42:41.975 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:42:41.975 INFO:tasks.workunit.client.0.smithi155.stdout: "features": { 2020-03-11T22:42:41.975 INFO:tasks.workunit.client.0.smithi155.stdout: "persistent": [ 2020-03-11T22:42:41.975 INFO:tasks.workunit.client.0.smithi155.stdout: "kraken", 2020-03-11T22:42:41.975 INFO:tasks.workunit.client.0.smithi155.stdout: "luminous", 2020-03-11T22:42:41.975 INFO:tasks.workunit.client.0.smithi155.stdout: "mimic", 2020-03-11T22:42:41.975 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap-prune", 2020-03-11T22:42:41.975 INFO:tasks.workunit.client.0.smithi155.stdout: "nautilus", 2020-03-11T22:42:41.976 INFO:tasks.workunit.client.0.smithi155.stdout: "octopus" 2020-03-11T22:42:41.976 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:41.976 INFO:tasks.workunit.client.0.smithi155.stdout: "optional": [] 2020-03-11T22:42:41.976 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.976 INFO:tasks.workunit.client.0.smithi155.stdout: "mons": [ 2020-03-11T22:42:41.976 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.976 INFO:tasks.workunit.client.0.smithi155.stdout: "rank": 0, 2020-03-11T22:42:41.977 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "a", 2020-03-11T22:42:41.977 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:42:41.977 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:41.977 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.977 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:41.977 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:42:41.978 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 0 2020-03-11T22:42:41.978 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:41.978 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:41.978 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.978 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:42:41.978 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:42:41.978 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 0, 2020-03-11T22:42:41.978 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 0 2020-03-11T22:42:41.979 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:41.979 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:41.979 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.979 INFO:tasks.workunit.client.0.smithi155.stdout: "quorum": [ 2020-03-11T22:42:41.979 INFO:tasks.workunit.client.0.smithi155.stdout: 0 2020-03-11T22:42:41.979 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:41.979 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap": { 2020-03-11T22:42:41.980 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 42, 2020-03-11T22:42:41.980 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:42:41.980 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:40:28.549981+0000", 2020-03-11T22:42:41.980 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:41:34.616654+0000", 2020-03-11T22:42:41.980 INFO:tasks.workunit.client.0.smithi155.stdout: "last_up_change": "2020-03-11T22:41:30.351578+0000", 2020-03-11T22:42:41.980 INFO:tasks.workunit.client.0.smithi155.stdout: "last_in_change": "2020-03-11T22:41:31.709833+0000", 2020-03-11T22:42:41.980 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:42:41.981 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_num": 5800384, 2020-03-11T22:42:41.981 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_set": [ 2020-03-11T22:42:41.981 INFO:tasks.workunit.client.0.smithi155.stdout: "nodown", 2020-03-11T22:42:41.981 INFO:tasks.workunit.client.0.smithi155.stdout: "noin", 2020-03-11T22:42:41.981 INFO:tasks.workunit.client.0.smithi155.stdout: "noout", 2020-03-11T22:42:41.981 INFO:tasks.workunit.client.0.smithi155.stdout: "pglog_hardlimit", 2020-03-11T22:42:41.981 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snapdirs", 2020-03-11T22:42:41.982 INFO:tasks.workunit.client.0.smithi155.stdout: "recovery_deletes", 2020-03-11T22:42:41.982 INFO:tasks.workunit.client.0.smithi155.stdout: "sortbitwise" 2020-03-11T22:42:41.982 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:41.982 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_version": 7, 2020-03-11T22:42:41.982 INFO:tasks.workunit.client.0.smithi155.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:42:41.982 INFO:tasks.workunit.client.0.smithi155.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:42:41.982 INFO:tasks.workunit.client.0.smithi155.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:42:41.982 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_snapshot": "", 2020-03-11T22:42:41.983 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_max": 1, 2020-03-11T22:42:41.983 INFO:tasks.workunit.client.0.smithi155.stdout: "max_osd": 3, 2020-03-11T22:42:41.983 INFO:tasks.workunit.client.0.smithi155.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:42:41.983 INFO:tasks.workunit.client.0.smithi155.stdout: "min_compat_client": "jewel", 2020-03-11T22:42:41.983 INFO:tasks.workunit.client.0.smithi155.stdout: "require_osd_release": "octopus", 2020-03-11T22:42:41.983 INFO:tasks.workunit.client.0.smithi155.stdout: "pools": [ 2020-03-11T22:42:41.984 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.984 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:42:41.984 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_name": "test", 2020-03-11T22:42:41.984 INFO:tasks.workunit.client.0.smithi155.stdout: "create_time": "2020-03-11T22:40:49.840683+0000", 2020-03-11T22:42:41.984 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": 1, 2020-03-11T22:42:41.984 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_names": "hashpspool", 2020-03-11T22:42:41.984 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2020-03-11T22:42:41.985 INFO:tasks.workunit.client.0.smithi155.stdout: "size": 3, 2020-03-11T22:42:41.985 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 2, 2020-03-11T22:42:41.985 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_rule": 0, 2020-03-11T22:42:41.985 INFO:tasks.workunit.client.0.smithi155.stdout: "object_hash": 2, 2020-03-11T22:42:41.985 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_autoscale_mode": "on", 2020-03-11T22:42:41.985 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num": 32, 2020-03-11T22:42:41.985 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num": 1, 2020-03-11T22:42:41.986 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num_target": 32, 2020-03-11T22:42:41.986 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_target": 32, 2020-03-11T22:42:41.986 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_pending": 32, 2020-03-11T22:42:41.986 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pg_merge_meta": { 2020-03-11T22:42:41.986 INFO:tasks.workunit.client.0.smithi155.stdout: "source_pgid": "0.0", 2020-03-11T22:42:41.986 INFO:tasks.workunit.client.0.smithi155.stdout: "ready_epoch": 0, 2020-03-11T22:42:41.986 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_started": 0, 2020-03-11T22:42:41.987 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_clean": 0, 2020-03-11T22:42:41.987 INFO:tasks.workunit.client.0.smithi155.stdout: "source_version": "0'0", 2020-03-11T22:42:41.987 INFO:tasks.workunit.client.0.smithi155.stdout: "target_version": "0'0" 2020-03-11T22:42:41.987 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.987 INFO:tasks.workunit.client.0.smithi155.stdout: "last_change": "41", 2020-03-11T22:42:41.987 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend": "0", 2020-03-11T22:42:41.987 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:42:41.988 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_preluminous": "41", 2020-03-11T22:42:41.988 INFO:tasks.workunit.client.0.smithi155.stdout: "auid": 0, 2020-03-11T22:42:41.988 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_mode": "selfmanaged", 2020-03-11T22:42:41.988 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_seq": 0, 2020-03-11T22:42:41.988 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_epoch": 0, 2020-03-11T22:42:41.988 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_snaps": [], 2020-03-11T22:42:41.988 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps": "[]", 2020-03-11T22:42:41.988 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_bytes": 0, 2020-03-11T22:42:41.989 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_objects": 0, 2020-03-11T22:42:41.989 INFO:tasks.workunit.client.0.smithi155.stdout: "tiers": [], 2020-03-11T22:42:41.989 INFO:tasks.workunit.client.0.smithi155.stdout: "tier_of": -1, 2020-03-11T22:42:41.989 INFO:tasks.workunit.client.0.smithi155.stdout: "read_tier": -1, 2020-03-11T22:42:41.989 INFO:tasks.workunit.client.0.smithi155.stdout: "write_tier": -1, 2020-03-11T22:42:41.989 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_mode": "none", 2020-03-11T22:42:41.989 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_bytes": 0, 2020-03-11T22:42:41.990 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_objects": 0, 2020-03-11T22:42:41.990 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:42:41.990 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:42:41.990 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:42:41.990 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_flush_age": 0, 2020-03-11T22:42:41.990 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_evict_age": 0, 2020-03-11T22:42:41.990 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profile": "", 2020-03-11T22:42:41.990 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_params": { 2020-03-11T22:42:41.991 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "none" 2020-03-11T22:42:41.991 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.991 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_period": 0, 2020-03-11T22:42:41.991 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_count": 0, 2020-03-11T22:42:41.991 INFO:tasks.workunit.client.0.smithi155.stdout: "use_gmt_hitset": true, 2020-03-11T22:42:41.991 INFO:tasks.workunit.client.0.smithi155.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:42:41.991 INFO:tasks.workunit.client.0.smithi155.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:42:41.992 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:42:41.992 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:42:41.992 INFO:tasks.workunit.client.0.smithi155.stdout: "grade_table": [], 2020-03-11T22:42:41.992 INFO:tasks.workunit.client.0.smithi155.stdout: "stripe_width": 0, 2020-03-11T22:42:41.992 INFO:tasks.workunit.client.0.smithi155.stdout: "expected_num_objects": 0, 2020-03-11T22:42:41.992 INFO:tasks.workunit.client.0.smithi155.stdout: "fast_read": false, 2020-03-11T22:42:41.992 INFO:tasks.workunit.client.0.smithi155.stdout: "options": {}, 2020-03-11T22:42:41.992 INFO:tasks.workunit.client.0.smithi155.stdout: "application_metadata": {} 2020-03-11T22:42:41.993 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:41.993 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:41.993 INFO:tasks.workunit.client.0.smithi155.stdout: "osds": [ 2020-03-11T22:42:41.993 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.993 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:42:41.993 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "53c048ef-1ae8-4c15-b86c-b15481d0a054", 2020-03-11T22:42:41.993 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:42:41.994 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:42:41.994 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:42:41.994 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:42:41.994 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 5, 2020-03-11T22:42:41.994 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 29, 2020-03-11T22:42:41.994 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 35, 2020-03-11T22:42:41.994 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 41, 2020-03-11T22:42:41.994 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 30, 2020-03-11T22:42:41.995 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:42:41.995 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:42:41.995 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:41.995 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.995 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:41.995 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:42:41.995 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:41.996 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.996 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.996 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:41.996 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:42:41.996 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:41.996 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:41.996 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:41.996 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.997 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:42:41.997 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:41.997 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.997 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:41.997 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:42:41.997 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:41.997 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.998 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.998 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:41.998 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:42:41.998 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:41.998 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:41.998 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:41.998 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.998 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:41.999 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:41.999 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:41.999 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:41.999 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:42:41.999 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:41.999 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:41.999 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.000 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:42.000 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:42:42.000 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:42.000 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.000 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.000 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.000 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:42.000 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:42.001 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.001 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:42.001 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:42:42.002 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:42.002 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.003 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.003 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:42.003 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:42:42.003 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39376 2020-03-11T22:42:42.003 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.003 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.004 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.004 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6803/39376", 2020-03-11T22:42:42.004 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6805/39376", 2020-03-11T22:42:42.004 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6809/39376", 2020-03-11T22:42:42.004 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6807/39376", 2020-03-11T22:42:42.004 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:42:42.004 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:42:42.005 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:42:42.005 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.005 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.005 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.005 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:42:42.005 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "ce1120e9-d62c-42de-b0b2-a04b6602228c", 2020-03-11T22:42:42.006 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 0, 2020-03-11T22:42:42.006 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:42:42.006 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:42:42.006 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:42:42.006 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 0, 2020-03-11T22:42:42.006 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 0, 2020-03-11T22:42:42.006 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 9, 2020-03-11T22:42:42.007 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 21, 2020-03-11T22:42:42.007 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 31, 2020-03-11T22:42:42.007 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:42:42.007 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:42:42.007 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:42.007 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.008 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:42.008 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:42:42.008 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:42.008 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.008 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.008 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:42.009 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:42:42.009 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:42.009 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.009 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.009 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.010 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:42:42.010 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:42.011 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.011 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:42.011 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:42:42.011 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:42.011 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.011 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.011 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:42.012 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:42:42.012 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:42.012 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.012 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.012 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.012 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:42.012 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:42.013 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.013 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:42.013 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:42:42.013 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:42.013 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.013 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.013 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:42.013 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:42:42.014 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:42.014 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.014 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.014 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.014 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:42.014 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:42.014 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.014 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:42.015 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:42:42.015 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:42.015 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.015 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.015 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:42.015 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:42:42.015 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 34960 2020-03-11T22:42:42.016 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.016 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.016 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.016 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/34960", 2020-03-11T22:42:42.016 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/34960", 2020-03-11T22:42:42.016 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/34960", 2020-03-11T22:42:42.016 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/34960", 2020-03-11T22:42:42.016 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:42:42.017 INFO:tasks.workunit.client.0.smithi155.stdout: "exists" 2020-03-11T22:42:42.017 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.017 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.017 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.017 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:42:42.017 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "8a4738de-77ce-4790-b3b3-87d310c2cbe9", 2020-03-11T22:42:42.017 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:42:42.017 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:42:42.018 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:42:42.018 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:42:42.018 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 14, 2020-03-11T22:42:42.018 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 29, 2020-03-11T22:42:42.018 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 36, 2020-03-11T22:42:42.018 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 0, 2020-03-11T22:42:42.019 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 32, 2020-03-11T22:42:42.019 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:42:42.019 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:42:42.020 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:42.020 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.020 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:42.020 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:42:42.020 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:42.020 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.021 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.021 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:42.021 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:42:42.021 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:42.021 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.021 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.021 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.022 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:42:42.022 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:42.022 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.022 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:42.022 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:42:42.022 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:42.022 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.022 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.023 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:42.023 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:42:42.023 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:42.023 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.023 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.023 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.023 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:42:42.023 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:42.024 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.024 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:42.024 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:42:42.024 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:42.024 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.024 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.024 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:42.025 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:42:42.025 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:42.025 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.025 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.025 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.025 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:42:42.025 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:42:42.025 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.026 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:42:42.026 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:42:42.026 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:42.026 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.026 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.026 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:42:42.026 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:42:42.026 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 39570 2020-03-11T22:42:42.027 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.027 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.027 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.027 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/39570", 2020-03-11T22:42:42.027 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/39570", 2020-03-11T22:42:42.027 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/39570", 2020-03-11T22:42:42.027 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/39570", 2020-03-11T22:42:42.028 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:42:42.028 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:42:42.028 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:42:42.028 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.028 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.028 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:42.028 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_xinfo": [ 2020-03-11T22:42:42.028 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.029 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:42:42.029 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:41:24.154203+0000", 2020-03-11T22:42:42.029 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:42:42.029 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:42:42.029 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:42:42.029 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:42:42.029 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:35.219964+0000", 2020-03-11T22:42:42.030 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:42:42.030 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.030 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.030 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:42:42.030 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:41:24.661922+0000", 2020-03-11T22:42:42.031 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:42:42.031 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:42:42.031 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:42:42.031 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:42:42.031 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:41.515052+0000", 2020-03-11T22:42:42.031 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:42:42.031 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.032 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.032 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:42:42.032 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:41:25.163731+0000", 2020-03-11T22:42:42.032 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:42:42.032 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:42:42.032 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:42:42.032 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:42:42.032 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:40:47.907713+0000", 2020-03-11T22:42:42.033 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:42:42.033 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.033 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:42.033 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap": [], 2020-03-11T22:42:42.033 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap_items": [], 2020-03-11T22:42:42.033 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_temp": [], 2020-03-11T22:42:42.033 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_temp": [], 2020-03-11T22:42:42.034 INFO:tasks.workunit.client.0.smithi155.stdout: "blacklist": {}, 2020-03-11T22:42:42.034 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profiles": { 2020-03-11T22:42:42.034 INFO:tasks.workunit.client.0.smithi155.stdout: "default": { 2020-03-11T22:42:42.034 INFO:tasks.workunit.client.0.smithi155.stdout: "k": "2", 2020-03-11T22:42:42.034 INFO:tasks.workunit.client.0.smithi155.stdout: "m": "2", 2020-03-11T22:42:42.034 INFO:tasks.workunit.client.0.smithi155.stdout: "plugin": "jerasure", 2020-03-11T22:42:42.034 INFO:tasks.workunit.client.0.smithi155.stdout: "technique": "reed_sol_van" 2020-03-11T22:42:42.035 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.035 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.035 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps_queue": [], 2020-03-11T22:42:42.035 INFO:tasks.workunit.client.0.smithi155.stdout: "new_removed_snaps": [], 2020-03-11T22:42:42.035 INFO:tasks.workunit.client.0.smithi155.stdout: "new_purged_snaps": [], 2020-03-11T22:42:42.035 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_node_flags": {}, 2020-03-11T22:42:42.035 INFO:tasks.workunit.client.0.smithi155.stdout: "device_class_flags": {} 2020-03-11T22:42:42.036 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.036 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_metadata": [ 2020-03-11T22:42:42.036 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.036 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:42:42.036 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:42:42.036 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6804/39376,v1:127.0.0.1:6805/39376]", 2020-03-11T22:42:42.037 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:42:42.037 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:42:42.037 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:42.037 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:42.037 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:42.037 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:42.037 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:42.038 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:42.038 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:42.038 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:42:42.038 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:42.038 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:42.038 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:42.038 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:42.038 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:42:42.039 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:42.039 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:42.039 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:42.039 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:42:42.039 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:42:42.039 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:42.039 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:42:42.040 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:42:42.040 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:42.040 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:42:42.040 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6802/39376,v1:127.0.0.1:6803/39376]", 2020-03-11T22:42:42.040 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:42:42.041 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/39376,v1:127.0.0.1:6809/39376]", 2020-03-11T22:42:42.041 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/39376,v1:127.0.0.1:6807/39376]", 2020-03-11T22:42:42.041 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:42:42.041 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:42:42.041 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:42.041 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:42.041 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:42.042 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:42:42.042 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:42.042 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:42.042 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:42:42.042 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:42:42.042 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:42.042 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:42:42.043 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.043 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.043 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:42:42.043 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:42:42.043 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/34960,v1:127.0.0.1:6813/34960]", 2020-03-11T22:42:42.043 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:42:42.043 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:42:42.043 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:42.044 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:42.044 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:42.044 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:42.044 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:42.044 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:42.044 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:42.044 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:42:42.044 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:42.045 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:42.045 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:42.045 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:42.045 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:42:42.045 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:42.045 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:42.045 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:42.045 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:42:42.046 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:42:42.046 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:42.046 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:42:42.046 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:42:42.046 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:42.046 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:42:42.048 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/34960,v1:127.0.0.1:6811/34960]", 2020-03-11T22:42:42.048 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:42:42.048 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/34960,v1:127.0.0.1:6817/34960]", 2020-03-11T22:42:42.048 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/34960,v1:127.0.0.1:6815/34960]", 2020-03-11T22:42:42.049 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:42:42.049 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:42:42.049 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:42.049 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:42.049 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:42.049 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:42:42.049 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:42.049 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:42.050 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:42:42.050 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:42:42.050 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:42.050 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:42:42.050 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.050 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.051 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:42:42.051 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:42:42.051 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/39570,v1:127.0.0.1:6813/39570]", 2020-03-11T22:42:42.051 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:42:42.051 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:42:42.052 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:42:42.052 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:42:42.052 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:42:42.052 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:42:42.052 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:42:42.052 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:42:42.052 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:42:42.052 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:42:42.053 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:42:42.053 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:42:42.053 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:42:42.053 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:42:42.053 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:42:42.053 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:42:42.053 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:42:42.053 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:42:42.054 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:42:42.054 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:42:42.054 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:42:42.054 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:42:42.054 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:42:42.054 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:42:42.054 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:42:42.055 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/39570,v1:127.0.0.1:6811/39570]", 2020-03-11T22:42:42.055 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:42:42.055 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/39570,v1:127.0.0.1:6817/39570]", 2020-03-11T22:42:42.055 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/39570,v1:127.0.0.1:6815/39570]", 2020-03-11T22:42:42.055 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:42:42.055 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:42:42.055 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:42:42.055 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:42:42.056 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:42:42.056 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:42:42.056 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:42:42.056 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:42:42.056 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:42:42.056 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:42:42.056 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:42:42.056 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:42:42.057 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.057 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:42.057 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_first_committed": 1, 2020-03-11T22:42:42.057 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_last_committed": 42, 2020-03-11T22:42:42.057 INFO:tasks.workunit.client.0.smithi155.stdout: "crushmap": { 2020-03-11T22:42:42.057 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": [ 2020-03-11T22:42:42.057 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.058 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:42:42.058 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.0", 2020-03-11T22:42:42.058 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:42:42.058 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.058 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.058 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:42:42.058 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.1", 2020-03-11T22:42:42.058 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:42:42.059 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.059 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.059 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:42:42.059 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.2", 2020-03-11T22:42:42.059 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:42:42.059 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.059 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:42.059 INFO:tasks.workunit.client.0.smithi155.stdout: "types": [ 2020-03-11T22:42:42.060 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.060 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 0, 2020-03-11T22:42:42.060 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd" 2020-03-11T22:42:42.060 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.060 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.060 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:42:42.060 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "host" 2020-03-11T22:42:42.060 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.061 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.061 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 2, 2020-03-11T22:42:42.061 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "chassis" 2020-03-11T22:42:42.061 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.061 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.062 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 3, 2020-03-11T22:42:42.062 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "rack" 2020-03-11T22:42:42.062 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.062 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.062 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 4, 2020-03-11T22:42:42.062 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "row" 2020-03-11T22:42:42.062 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.063 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.063 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 5, 2020-03-11T22:42:42.063 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pdu" 2020-03-11T22:42:42.063 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.063 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.063 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 6, 2020-03-11T22:42:42.063 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pod" 2020-03-11T22:42:42.063 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.064 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.064 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 7, 2020-03-11T22:42:42.064 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "room" 2020-03-11T22:42:42.064 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.064 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.064 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 8, 2020-03-11T22:42:42.064 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "datacenter" 2020-03-11T22:42:42.064 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.065 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.065 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 9, 2020-03-11T22:42:42.065 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "zone" 2020-03-11T22:42:42.065 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.065 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.065 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 10, 2020-03-11T22:42:42.065 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "region" 2020-03-11T22:42:42.065 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.066 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.066 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:42:42.066 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "root" 2020-03-11T22:42:42.066 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.066 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:42.066 INFO:tasks.workunit.client.0.smithi155.stdout: "buckets": [ 2020-03-11T22:42:42.066 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.066 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -1, 2020-03-11T22:42:42.067 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default", 2020-03-11T22:42:42.067 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:42:42.067 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:42:42.067 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:42:42.067 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:42:42.067 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:42:42.067 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:42:42.067 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.068 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:42:42.068 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:42:42.068 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:42:42.068 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.068 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.068 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.068 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.068 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -2, 2020-03-11T22:42:42.069 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default~hdd", 2020-03-11T22:42:42.069 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:42:42.069 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:42:42.069 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:42:42.069 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:42:42.069 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:42:42.069 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:42:42.070 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.070 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:42:42.070 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:42:42.070 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:42:42.070 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.070 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.070 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.070 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.071 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:42:42.071 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155", 2020-03-11T22:42:42.071 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:42:42.071 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:42:42.071 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:42:42.071 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:42:42.072 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:42:42.072 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:42:42.072 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.072 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:42:42.072 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:42:42.072 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:42:42.073 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.073 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.073 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:42:42.073 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:42:42.073 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:42:42.073 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.073 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.073 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:42:42.074 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:42:42.074 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:42:42.074 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.074 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.074 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.074 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.074 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:42:42.074 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155~hdd", 2020-03-11T22:42:42.075 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:42:42.075 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:42:42.075 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:42:42.075 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:42:42.075 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:42:42.075 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:42:42.075 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.076 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:42:42.076 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:42:42.076 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:42:42.076 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.076 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.076 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:42:42.076 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:42:42.076 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:42:42.077 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.077 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.077 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:42:42.077 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:42:42.077 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:42:42.077 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.077 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.078 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.078 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:42.078 INFO:tasks.workunit.client.0.smithi155.stdout: "rules": [ 2020-03-11T22:42:42.078 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.078 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_id": 0, 2020-03-11T22:42:42.078 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_name": "replicated_rule", 2020-03-11T22:42:42.078 INFO:tasks.workunit.client.0.smithi155.stdout: "ruleset": 0, 2020-03-11T22:42:42.078 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2020-03-11T22:42:42.079 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 1, 2020-03-11T22:42:42.079 INFO:tasks.workunit.client.0.smithi155.stdout: "max_size": 10, 2020-03-11T22:42:42.079 INFO:tasks.workunit.client.0.smithi155.stdout: "steps": [ 2020-03-11T22:42:42.079 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.079 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "take", 2020-03-11T22:42:42.079 INFO:tasks.workunit.client.0.smithi155.stdout: "item": -1, 2020-03-11T22:42:42.079 INFO:tasks.workunit.client.0.smithi155.stdout: "item_name": "default" 2020-03-11T22:42:42.079 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.080 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.080 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "choose_firstn", 2020-03-11T22:42:42.080 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 0, 2020-03-11T22:42:42.080 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "osd" 2020-03-11T22:42:42.080 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.080 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.080 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "emit" 2020-03-11T22:42:42.080 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.081 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:42:42.081 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.081 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:42.081 INFO:tasks.workunit.client.0.smithi155.stdout: "tunables": { 2020-03-11T22:42:42.081 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_tries": 0, 2020-03-11T22:42:42.081 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:42:42.081 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_total_tries": 50, 2020-03-11T22:42:42.082 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:42:42.082 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:42:42.082 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_stable": 1, 2020-03-11T22:42:42.082 INFO:tasks.workunit.client.0.smithi155.stdout: "straw_calc_version": 1, 2020-03-11T22:42:42.082 INFO:tasks.workunit.client.0.smithi155.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:42:42.083 INFO:tasks.workunit.client.0.smithi155.stdout: "profile": "jewel", 2020-03-11T22:42:42.083 INFO:tasks.workunit.client.0.smithi155.stdout: "optimal_tunables": 1, 2020-03-11T22:42:42.083 INFO:tasks.workunit.client.0.smithi155.stdout: "legacy_tunables": 0, 2020-03-11T22:42:42.083 INFO:tasks.workunit.client.0.smithi155.stdout: "minimum_required_version": "jewel", 2020-03-11T22:42:42.083 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables": 1, 2020-03-11T22:42:42.083 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables2": 1, 2020-03-11T22:42:42.083 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v2_rules": 0, 2020-03-11T22:42:42.083 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables3": 1, 2020-03-11T22:42:42.084 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v3_rules": 0, 2020-03-11T22:42:42.084 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v4_buckets": 1, 2020-03-11T22:42:42.084 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables5": 1, 2020-03-11T22:42:42.084 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v5_rules": 0 2020-03-11T22:42:42.084 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.084 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_args": {} 2020-03-11T22:42:42.084 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.084 INFO:tasks.workunit.client.0.smithi155.stdout: "fsmap": { 2020-03-11T22:42:42.085 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:42:42.085 INFO:tasks.workunit.client.0.smithi155.stdout: "default_fscid": -1, 2020-03-11T22:42:42.085 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": { 2020-03-11T22:42:42.085 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": {}, 2020-03-11T22:42:42.085 INFO:tasks.workunit.client.0.smithi155.stdout: "ro_compat": {}, 2020-03-11T22:42:42.085 INFO:tasks.workunit.client.0.smithi155.stdout: "incompat": { 2020-03-11T22:42:42.085 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_1": "base v0.20", 2020-03-11T22:42:42.086 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:42:42.086 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:42:42.086 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:42:42.086 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:42:42.086 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:42:42.086 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_8": "no anchor table", 2020-03-11T22:42:42.086 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_9": "file layout v2", 2020-03-11T22:42:42.086 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:42:42.087 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.087 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.087 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_flags": { 2020-03-11T22:42:42.087 INFO:tasks.workunit.client.0.smithi155.stdout: "enable_multiple": false, 2020-03-11T22:42:42.087 INFO:tasks.workunit.client.0.smithi155.stdout: "ever_enabled_multiple": false 2020-03-11T22:42:42.087 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.087 INFO:tasks.workunit.client.0.smithi155.stdout: "standbys": [], 2020-03-11T22:42:42.087 INFO:tasks.workunit.client.0.smithi155.stdout: "filesystems": [] 2020-03-11T22:42:42.088 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.088 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:42:42.088 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:42:42.088 INFO:tasks.workunit.client.0.smithi155.stdout: "auth": { 2020-03-11T22:42:42.088 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:42:42.088 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 5, 2020-03-11T22:42:42.088 INFO:tasks.workunit.client.0.smithi155.stdout: "num_secrets": 3 2020-03-11T22:42:42.088 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.089 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 32, 2020-03-11T22:42:42.089 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_active": 32, 2020-03-11T22:42:42.089 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_unknown": 0, 2020-03-11T22:42:42.089 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osd": 3, 2020-03-11T22:42:42.089 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_sum": { 2020-03-11T22:42:42.089 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:42:42.089 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 47947, 2020-03-11T22:42:42.089 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 100, 2020-03-11T22:42:42.090 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2020-03-11T22:42:42.090 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 300, 2020-03-11T22:42:42.090 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:42:42.090 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:42:42.090 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 100, 2020-03-11T22:42:42.090 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:42:42.090 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:42:42.091 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 100, 2020-03-11T22:42:42.091 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:42:42.091 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:42:42.091 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:42:42.091 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 101, 2020-03-11T22:42:42.091 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 101, 2020-03-11T22:42:42.091 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:42:42.091 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:42:42.092 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:42:42.092 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:42:42.092 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:42:42.093 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:42:42.093 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:42:42.093 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:42:42.093 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:42:42.093 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:42:42.093 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:42:42.093 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:42:42.094 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:42:42.094 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:42:42.094 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:42:42.094 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:42:42.094 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:42:42.094 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:42:42.094 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:42:42.095 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:42:42.095 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:42:42.095 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:42:42.095 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:42:42.095 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:42:42.095 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:42:42.095 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.095 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:42:42.096 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:42:42.096 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:42:42.096 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:42:42.096 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 0, 2020-03-11T22:42:42.096 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 0, 2020-03-11T22:42:42.096 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:42:42.096 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:42.096 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:42:42.097 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:42:42.097 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:42:42.097 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.097 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 513, 2020-03-11T22:42:42.097 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 513, 2020-03-11T22:42:42.097 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 64, 2020-03-11T22:42:42.097 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 64, 2020-03-11T22:42:42.098 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 0 2020-03-11T22:42:42.098 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.098 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum": { 2020-03-11T22:42:42.098 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:42:42.098 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:42:42.098 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 65, 2020-03-11T22:42:42.098 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:42:42.098 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:42:42.099 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:42:42.099 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:42:42.099 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3179968, 2020-03-11T22:42:42.099 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 33920, 2020-03-11T22:42:42.099 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 14, 2020-03-11T22:42:42.099 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145713, 2020-03-11T22:42:42.099 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311392832, 2020-03-11T22:42:42.099 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:42:42.100 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:42:42.100 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318866259968, 2020-03-11T22:42:42.100 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:42:42.100 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 34734080, 2020-03-11T22:42:42.100 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 427025, 2020-03-11T22:42:42.100 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:42:42.100 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:42.101 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:42:42.101 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 14646, 2020-03-11T22:42:42.101 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221210826 2020-03-11T22:42:42.101 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.101 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:42:42.101 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:42:42.101 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:42:42.101 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2020-03-11T22:42:42.102 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:42:42.102 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:42:42.102 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:42:42.102 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.102 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:42:42.102 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:42:42.102 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:42:42.102 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:42:42.103 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:42:42.103 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.103 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:42:42.103 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:42:42.104 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.104 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum_by_class": { 2020-03-11T22:42:42.104 INFO:tasks.workunit.client.0.smithi155.stdout: "hdd": { 2020-03-11T22:42:42.104 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:42:42.104 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:42:42.104 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 65, 2020-03-11T22:42:42.104 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:42:42.104 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:42:42.105 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:42:42.105 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:42:42.105 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3179968, 2020-03-11T22:42:42.105 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 33920, 2020-03-11T22:42:42.105 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 14, 2020-03-11T22:42:42.105 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145713, 2020-03-11T22:42:42.105 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311392832, 2020-03-11T22:42:42.106 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:42:42.106 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:42:42.106 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318866259968, 2020-03-11T22:42:42.106 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:42:42.106 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 34734080, 2020-03-11T22:42:42.106 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 427025, 2020-03-11T22:42:42.106 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:42:42.106 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:42.107 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:42:42.107 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 14646, 2020-03-11T22:42:42.107 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221210826 2020-03-11T22:42:42.107 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.107 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:42:42.107 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:42:42.107 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:42:42.107 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2020-03-11T22:42:42.108 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:42:42.108 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:42:42.108 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:42:42.108 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.108 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:42:42.108 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:42:42.108 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:42:42.109 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:42:42.109 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:42:42.109 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.109 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:42:42.109 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:42:42.109 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.109 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.109 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_stats": [ 2020-03-11T22:42:42.110 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.110 INFO:tasks.workunit.client.0.smithi155.stdout: "poolid": 1, 2020-03-11T22:42:42.110 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 32, 2020-03-11T22:42:42.110 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:42:42.110 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 47947, 2020-03-11T22:42:42.110 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 100, 2020-03-11T22:42:42.110 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 0, 2020-03-11T22:42:42.110 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 300, 2020-03-11T22:42:42.111 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:42:42.111 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:42:42.111 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 100, 2020-03-11T22:42:42.111 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:42:42.111 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:42:42.111 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 100, 2020-03-11T22:42:42.111 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:42:42.112 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:42:42.112 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:42:42.112 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 101, 2020-03-11T22:42:42.112 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 101, 2020-03-11T22:42:42.112 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:42:42.113 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:42:42.113 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:42:42.113 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:42:42.113 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:42:42.114 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:42:42.114 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:42:42.114 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:42:42.114 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:42:42.114 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:42:42.114 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:42:42.114 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:42:42.115 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:42:42.115 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:42:42.115 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:42:42.115 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:42:42.115 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:42:42.115 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:42:42.115 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:42:42.115 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:42:42.116 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:42:42.116 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:42:42.116 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:42:42.116 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:42:42.116 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:42:42.116 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.116 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:42:42.117 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:42:42.117 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:42:42.117 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:42:42.117 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 19529728, 2020-03-11T22:42:42.117 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 142522, 2020-03-11T22:42:42.117 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:42:42.117 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:42:42.117 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:42:42.118 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:42:42.118 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:42:42.118 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.118 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 513, 2020-03-11T22:42:42.118 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 513, 2020-03-11T22:42:42.118 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 64, 2020-03-11T22:42:42.118 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 64, 2020-03-11T22:42:42.119 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 3 2020-03-11T22:42:42.119 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.119 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:42.119 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_stats": [ 2020-03-11T22:42:42.119 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.119 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:42:42.119 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 150323855379 2020-03-11T22:42:42.120 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.120 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.120 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:42:42.120 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 38654705679 2020-03-11T22:42:42.120 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.120 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.120 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:42:42.120 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 154618822675 2020-03-11T22:42:42.121 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.121 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:42.121 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_state": [ 2020-03-11T22:42:42.121 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.121 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "active+undersized+degraded", 2020-03-11T22:42:42.121 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 32 2020-03-11T22:42:42.122 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.122 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:42.122 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_osd": [ 2020-03-11T22:42:42.122 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.122 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:42:42.122 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:42:42.122 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 32, 2020-03-11T22:42:42.122 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:42.123 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.123 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.123 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:42:42.123 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:42:42.123 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 0, 2020-03-11T22:42:42.123 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:42.123 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.124 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.124 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:42:42.124 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 32, 2020-03-11T22:42:42.124 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 32, 2020-03-11T22:42:42.124 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:42:42.125 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.125 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:42.125 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": [ 2020-03-11T22:42:42.125 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:42:42.125 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:42:42.125 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": {} 2020-03-11T22:42:42.125 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.126 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:42:42.126 INFO:tasks.workunit.client.0.smithi155.stdout: "servicemap": { 2020-03-11T22:42:42.126 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:42:42.126 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:40:28.550306+0000", 2020-03-11T22:42:42.126 INFO:tasks.workunit.client.0.smithi155.stdout: "services": {} 2020-03-11T22:42:42.126 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:42:42.126 INFO:tasks.workunit.client.0.smithi155.stdout: "paxos": { 2020-03-11T22:42:42.126 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:42:42.127 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 387, 2020-03-11T22:42:42.127 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pn": 0, 2020-03-11T22:42:42.127 INFO:tasks.workunit.client.0.smithi155.stdout: "accepted_pn": 0 2020-03-11T22:42:42.127 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:42:42.127 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:42:42.128 INFO:tasks.workunit.client.0.smithi155.stderr:report 460170817 2020-03-11T22:42:42.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:42:42.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:746: TEST_divergent_3: echo 'delay recovery' 2020-03-11T22:42:42.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:747: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:42.129 INFO:tasks.workunit.client.0.smithi155.stdout:delay recovery 2020-03-11T22:42:42.276 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:42.276 INFO:tasks.workunit.client.0.smithi155.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.929055+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.276 INFO:tasks.workunit.client.0.smithi155.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928394+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.276 INFO:tasks.workunit.client.0.smithi155.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928671+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.277 INFO:tasks.workunit.client.0.smithi155.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928362+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.277 INFO:tasks.workunit.client.0.smithi155.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928831+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.277 INFO:tasks.workunit.client.0.smithi155.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926198+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.277 INFO:tasks.workunit.client.0.smithi155.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926228+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.277 INFO:tasks.workunit.client.0.smithi155.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928059+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.277 INFO:tasks.workunit.client.0.smithi155.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927991+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.277 INFO:tasks.workunit.client.0.smithi155.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928167+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.278 INFO:tasks.workunit.client.0.smithi155.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927262+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.278 INFO:tasks.workunit.client.0.smithi155.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926848+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.278 INFO:tasks.workunit.client.0.smithi155.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927687+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.278 INFO:tasks.workunit.client.0.smithi155.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927622+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.278 INFO:tasks.workunit.client.0.smithi155.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927785+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.278 INFO:tasks.workunit.client.0.smithi155.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927684+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.278 INFO:tasks.workunit.client.0.smithi155.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927453+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.278 INFO:tasks.workunit.client.0.smithi155.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927314+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.279 INFO:tasks.workunit.client.0.smithi155.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926849+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.279 INFO:tasks.workunit.client.0.smithi155.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927401+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.279 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:48.076964+0000 29'100 42:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.279 INFO:tasks.workunit.client.0.smithi155.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928094+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.279 INFO:tasks.workunit.client.0.smithi155.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928720+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.279 INFO:tasks.workunit.client.0.smithi155.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926387+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.279 INFO:tasks.workunit.client.0.smithi155.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926235+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.280 INFO:tasks.workunit.client.0.smithi155.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926193+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.280 INFO:tasks.workunit.client.0.smithi155.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.929022+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.280 INFO:tasks.workunit.client.0.smithi155.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926841+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.280 INFO:tasks.workunit.client.0.smithi155.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926989+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.280 INFO:tasks.workunit.client.0.smithi155.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927241+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.280 INFO:tasks.workunit.client.0.smithi155.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:34.926847+0000 42'101 42:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.280 INFO:tasks.workunit.client.0.smithi155.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927819+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.281 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:42:42.281 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:42.281 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:42:42.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:42:42.286 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.0 2020-03-11T22:42:42.287 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:42:42.287 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:42:42.287 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:42.287 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:42.287 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:42:42.287 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.0.asok 2020-03-11T22:42:42.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:42:42.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:42:42.379 INFO:tasks.workunit.client.0.smithi155.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:42:42.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:748: TEST_divergent_3: for i in $non_divergent 2020-03-11T22:42:42.387 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: get_asok_path osd.2 2020-03-11T22:42:42.387 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:42:42.387 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:42:42.387 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:42.387 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:42.388 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:42:42.388 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.2.asok 2020-03-11T22:42:42.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:42:42.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:750: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:42:42.483 INFO:tasks.workunit.client.0.smithi155.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:42:42.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:754: TEST_divergent_3: echo 'revive divergent 1' 2020-03-11T22:42:42.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:755: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:42.487 INFO:tasks.workunit.client.0.smithi155.stdout:revive divergent 1 2020-03-11T22:42:42.810 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:42.810 INFO:tasks.workunit.client.0.smithi155.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.929055+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.810 INFO:tasks.workunit.client.0.smithi155.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928394+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.811 INFO:tasks.workunit.client.0.smithi155.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928671+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.811 INFO:tasks.workunit.client.0.smithi155.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928362+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.811 INFO:tasks.workunit.client.0.smithi155.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928831+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.811 INFO:tasks.workunit.client.0.smithi155.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926198+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.811 INFO:tasks.workunit.client.0.smithi155.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926228+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.811 INFO:tasks.workunit.client.0.smithi155.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928059+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.811 INFO:tasks.workunit.client.0.smithi155.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927991+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.812 INFO:tasks.workunit.client.0.smithi155.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928167+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.812 INFO:tasks.workunit.client.0.smithi155.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927262+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.812 INFO:tasks.workunit.client.0.smithi155.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926848+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.812 INFO:tasks.workunit.client.0.smithi155.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927687+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.812 INFO:tasks.workunit.client.0.smithi155.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927622+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.812 INFO:tasks.workunit.client.0.smithi155.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927785+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.813 INFO:tasks.workunit.client.0.smithi155.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927684+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.813 INFO:tasks.workunit.client.0.smithi155.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927453+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.813 INFO:tasks.workunit.client.0.smithi155.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927314+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.813 INFO:tasks.workunit.client.0.smithi155.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926849+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.813 INFO:tasks.workunit.client.0.smithi155.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927401+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.813 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:48.076964+0000 29'100 42:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.813 INFO:tasks.workunit.client.0.smithi155.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928094+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.814 INFO:tasks.workunit.client.0.smithi155.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928720+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.814 INFO:tasks.workunit.client.0.smithi155.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926387+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.814 INFO:tasks.workunit.client.0.smithi155.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926235+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.814 INFO:tasks.workunit.client.0.smithi155.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926193+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.814 INFO:tasks.workunit.client.0.smithi155.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.929022+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.814 INFO:tasks.workunit.client.0.smithi155.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926841+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.814 INFO:tasks.workunit.client.0.smithi155.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926989+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.814 INFO:tasks.workunit.client.0.smithi155.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927241+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.815 INFO:tasks.workunit.client.0.smithi155.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:34.926847+0000 42'101 42:130 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.815 INFO:tasks.workunit.client.0.smithi155.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927819+0000 29'100 42:129 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:42.815 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:42:42.815 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:42.815 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:42:42.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:756: TEST_divergent_3: ceph osd set noup 2020-03-11T22:42:43.484 INFO:tasks.workunit.client.0.smithi155.stderr:noup is set 2020-03-11T22:42:43.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:757: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:42:43.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:42:43.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:42:43.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:42:43.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:42:43.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:42:43.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:42:43.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:42:43.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:42:43.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:42:43.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:42:43.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:42:43.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:42:43.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:42:43.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:42:43.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:42:43.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:42:43.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:42:43.498 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:42:43.498 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:43.498 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:42:43.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:42:43.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:42:43.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:42:43.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:42:43.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:42:43.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:42:43.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:42:43.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:43.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:42:43.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:42:43.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:42:43.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:42:43.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:42:43.500 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:42:43.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:42:43.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:42:43.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:42:43.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:42:43.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:42:43.522 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:42:43.518+0000 7fb261cadec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:43.523 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:42:43.520+0000 7fb261cadec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:43.526 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:42:43.522+0000 7fb261cadec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:42:43.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:42:43.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:758: TEST_divergent_3: sleep 5 2020-03-11T22:42:44.041 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:42:44.038+0000 7fb261cadec0 -1 Falling back to public interface 2020-03-11T22:42:44.865 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:42:44.862+0000 7fb261cadec0 -1 osd.1 29 log_to_monitors {default=true} 2020-03-11T22:42:48.851 INFO:tasks.workunit.client.0.smithi155.stdout:delay recovery divergent 2020-03-11T22:42:48.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:760: TEST_divergent_3: echo 'delay recovery divergent' 2020-03-11T22:42:48.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:761: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:49.185 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:49.185 INFO:tasks.workunit.client.0.smithi155.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.929055+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.185 INFO:tasks.workunit.client.0.smithi155.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928394+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.185 INFO:tasks.workunit.client.0.smithi155.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928671+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.186 INFO:tasks.workunit.client.0.smithi155.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928362+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.186 INFO:tasks.workunit.client.0.smithi155.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928831+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.186 INFO:tasks.workunit.client.0.smithi155.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926198+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.186 INFO:tasks.workunit.client.0.smithi155.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926228+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.186 INFO:tasks.workunit.client.0.smithi155.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928059+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.186 INFO:tasks.workunit.client.0.smithi155.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927991+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.187 INFO:tasks.workunit.client.0.smithi155.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928167+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.187 INFO:tasks.workunit.client.0.smithi155.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927262+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.187 INFO:tasks.workunit.client.0.smithi155.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926848+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.187 INFO:tasks.workunit.client.0.smithi155.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927687+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.187 INFO:tasks.workunit.client.0.smithi155.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927622+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.187 INFO:tasks.workunit.client.0.smithi155.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927785+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.187 INFO:tasks.workunit.client.0.smithi155.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927684+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.188 INFO:tasks.workunit.client.0.smithi155.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927453+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.188 INFO:tasks.workunit.client.0.smithi155.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927314+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.188 INFO:tasks.workunit.client.0.smithi155.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926849+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.188 INFO:tasks.workunit.client.0.smithi155.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927401+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.188 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:48.076964+0000 29'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.188 INFO:tasks.workunit.client.0.smithi155.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928094+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.188 INFO:tasks.workunit.client.0.smithi155.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928720+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.189 INFO:tasks.workunit.client.0.smithi155.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926387+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.189 INFO:tasks.workunit.client.0.smithi155.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926235+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.189 INFO:tasks.workunit.client.0.smithi155.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926193+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.189 INFO:tasks.workunit.client.0.smithi155.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.929022+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.189 INFO:tasks.workunit.client.0.smithi155.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926841+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.189 INFO:tasks.workunit.client.0.smithi155.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926989+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.190 INFO:tasks.workunit.client.0.smithi155.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927241+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.190 INFO:tasks.workunit.client.0.smithi155.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:34.926847+0000 42'101 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.190 INFO:tasks.workunit.client.0.smithi155.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927819+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:49.190 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:42:49.190 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:49.191 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:42:49.195 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:42:49.196 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:42:49.196 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:42:49.196 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:42:49.196 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:42:49.196 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:42:49.196 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.1.asok 2020-03-11T22:42:49.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:42:49.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:762: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:42:49.287 INFO:tasks.workunit.client.0.smithi155.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:42:49.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:764: TEST_divergent_3: ceph osd unset noup 2020-03-11T22:42:50.151 INFO:tasks.workunit.client.0.smithi155.stderr:noup is unset 2020-03-11T22:42:50.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:766: TEST_divergent_3: wait_for_osd up 0 2020-03-11T22:42:50.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:50.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:42:50.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:50.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:50.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:50.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:50.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:50.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:42:50.161 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:42:50.549 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 35 up_thru 41 down_at 30 last_clean_interval [5,29) [v2:127.0.0.1:6802/39376,v1:127.0.0.1:6803/39376] [v2:127.0.0.1:6804/39376,v1:127.0.0.1:6805/39376] exists,up 53c048ef-1ae8-4c15-b86c-b15481d0a054 2020-03-11T22:42:50.549 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:42:50.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:50.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:50.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:50.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:767: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:42:50.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:50.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:42:50.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:50.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:50.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:50.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:50.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:50.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:42:50.890 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [9,29) [v2:127.0.0.1:6818/42118,v1:127.0.0.1:6819/42118] [v2:127.0.0.1:6820/42118,v1:127.0.0.1:6821/42118] exists,up ce1120e9-d62c-42de-b0b2-a04b6602228c 2020-03-11T22:42:50.890 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:42:50.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:50.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:50.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:50.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:768: TEST_divergent_3: wait_for_osd up 2 2020-03-11T22:42:50.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:42:50.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:42:50.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:42:50.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:42:50.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:42:50.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:42:50.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:42:50.892 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:42:51.232 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 36 up_thru 0 down_at 32 last_clean_interval [14,29) [v2:127.0.0.1:6810/39570,v1:127.0.0.1:6811/39570] [v2:127.0.0.1:6812/39570,v1:127.0.0.1:6813/39570] exists,up 8a4738de-77ce-4790-b3b3-87d310c2cbe9 2020-03-11T22:42:51.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:42:51.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:42:51.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:42:51.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:770: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:51.551 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:51.552 INFO:tasks.workunit.client.0.smithi155.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.929055+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.552 INFO:tasks.workunit.client.0.smithi155.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928394+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.552 INFO:tasks.workunit.client.0.smithi155.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928671+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.552 INFO:tasks.workunit.client.0.smithi155.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928362+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.552 INFO:tasks.workunit.client.0.smithi155.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928831+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.552 INFO:tasks.workunit.client.0.smithi155.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926198+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.553 INFO:tasks.workunit.client.0.smithi155.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926228+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.553 INFO:tasks.workunit.client.0.smithi155.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928059+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.553 INFO:tasks.workunit.client.0.smithi155.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927991+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.553 INFO:tasks.workunit.client.0.smithi155.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928167+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.553 INFO:tasks.workunit.client.0.smithi155.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927262+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.553 INFO:tasks.workunit.client.0.smithi155.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926848+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.553 INFO:tasks.workunit.client.0.smithi155.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927687+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.554 INFO:tasks.workunit.client.0.smithi155.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927622+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.554 INFO:tasks.workunit.client.0.smithi155.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927785+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.554 INFO:tasks.workunit.client.0.smithi155.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927684+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.554 INFO:tasks.workunit.client.0.smithi155.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927453+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.554 INFO:tasks.workunit.client.0.smithi155.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927314+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.554 INFO:tasks.workunit.client.0.smithi155.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926849+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.554 INFO:tasks.workunit.client.0.smithi155.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927401+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.554 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:48.076964+0000 29'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.555 INFO:tasks.workunit.client.0.smithi155.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928094+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.555 INFO:tasks.workunit.client.0.smithi155.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928720+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.555 INFO:tasks.workunit.client.0.smithi155.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926387+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.555 INFO:tasks.workunit.client.0.smithi155.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926235+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.555 INFO:tasks.workunit.client.0.smithi155.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926193+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.555 INFO:tasks.workunit.client.0.smithi155.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.929022+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.556 INFO:tasks.workunit.client.0.smithi155.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926841+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.556 INFO:tasks.workunit.client.0.smithi155.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926989+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.556 INFO:tasks.workunit.client.0.smithi155.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927241+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.556 INFO:tasks.workunit.client.0.smithi155.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:34.926847+0000 42'101 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.556 INFO:tasks.workunit.client.0.smithi155.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927819+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.556 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:42:51.556 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:51.557 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:42:51.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:771: TEST_divergent_3: echo 'wait for peering' 2020-03-11T22:42:51.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:772: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:42:51.561 INFO:tasks.workunit.client.0.smithi155.stdout:wait for peering 2020-03-11T22:42:51.881 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:42:51.881 INFO:tasks.workunit.client.0.smithi155.stdout:1.1f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.929055+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.881 INFO:tasks.workunit.client.0.smithi155.stdout:1.1e 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928394+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.882 INFO:tasks.workunit.client.0.smithi155.stdout:1.1d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928671+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.882 INFO:tasks.workunit.client.0.smithi155.stdout:1.1c 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928362+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.882 INFO:tasks.workunit.client.0.smithi155.stdout:1.1b 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928831+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.882 INFO:tasks.workunit.client.0.smithi155.stdout:1.1a 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926198+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.882 INFO:tasks.workunit.client.0.smithi155.stdout:1.19 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926228+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.882 INFO:tasks.workunit.client.0.smithi155.stdout:1.18 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928059+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.882 INFO:tasks.workunit.client.0.smithi155.stdout:1.17 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927991+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.882 INFO:tasks.workunit.client.0.smithi155.stdout:1.16 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928167+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.883 INFO:tasks.workunit.client.0.smithi155.stdout:1.15 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927262+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.883 INFO:tasks.workunit.client.0.smithi155.stdout:1.14 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926848+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.883 INFO:tasks.workunit.client.0.smithi155.stdout:1.13 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927687+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.883 INFO:tasks.workunit.client.0.smithi155.stdout:1.12 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927622+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.883 INFO:tasks.workunit.client.0.smithi155.stdout:1.11 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927785+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.883 INFO:tasks.workunit.client.0.smithi155.stdout:1.10 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927684+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.883 INFO:tasks.workunit.client.0.smithi155.stdout:1.5 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927453+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.883 INFO:tasks.workunit.client.0.smithi155.stdout:1.4 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927314+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.884 INFO:tasks.workunit.client.0.smithi155.stdout:1.3 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926849+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.884 INFO:tasks.workunit.client.0.smithi155.stdout:1.2 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927401+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.884 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:48.076964+0000 29'100 43:133 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.884 INFO:tasks.workunit.client.0.smithi155.stdout:1.1 4 0 4 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928094+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.884 INFO:tasks.workunit.client.0.smithi155.stdout:1.6 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.928720+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.884 INFO:tasks.workunit.client.0.smithi155.stdout:1.7 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926387+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.884 INFO:tasks.workunit.client.0.smithi155.stdout:1.8 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926235+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.884 INFO:tasks.workunit.client.0.smithi155.stdout:1.9 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926193+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.885 INFO:tasks.workunit.client.0.smithi155.stdout:1.a 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.929022+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.885 INFO:tasks.workunit.client.0.smithi155.stdout:1.b 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926841+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.885 INFO:tasks.workunit.client.0.smithi155.stdout:1.c 3 0 3 0 0 1510 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.926989+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.885 INFO:tasks.workunit.client.0.smithi155.stdout:1.d 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927241+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.885 INFO:tasks.workunit.client.0.smithi155.stdout:1.e 3 0 3 0 0 1156 0 0 17 17 active+undersized+degraded 2020-03-11T22:41:34.926847+0000 42'101 43:132 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.885 INFO:tasks.workunit.client.0.smithi155.stdout:1.f 3 0 3 0 0 1509 0 0 16 16 active+undersized+degraded 2020-03-11T22:41:34.927819+0000 29'100 43:131 [0,2] 0 [0,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:42:51.885 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:42:51.885 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:42:51.886 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:42:51.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:773: TEST_divergent_3: rados -p test put foo /etc/fstab 2020-03-11T22:43:02.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:777: TEST_divergent_3: echo 'killing divergent 1' 2020-03-11T22:43:02.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:778: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:02.957 INFO:tasks.workunit.client.0.smithi155.stdout:killing divergent 1 2020-03-11T22:43:03.288 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:03.288 INFO:tasks.workunit.client.0.smithi155.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.436618+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.288 INFO:tasks.workunit.client.0.smithi155.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.436347+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.289 INFO:tasks.workunit.client.0.smithi155.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.434596+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.289 INFO:tasks.workunit.client.0.smithi155.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.433720+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.289 INFO:tasks.workunit.client.0.smithi155.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:42:50.432702+0000 29'100 47:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.289 INFO:tasks.workunit.client.0.smithi155.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.434284+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.289 INFO:tasks.workunit.client.0.smithi155.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.433985+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.289 INFO:tasks.workunit.client.0.smithi155.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.434166+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.290 INFO:tasks.workunit.client.0.smithi155.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.433503+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.290 INFO:tasks.workunit.client.0.smithi155.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.432156+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.290 INFO:tasks.workunit.client.0.smithi155.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.435547+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.290 INFO:tasks.workunit.client.0.smithi155.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.434516+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.290 INFO:tasks.workunit.client.0.smithi155.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.433817+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.290 INFO:tasks.workunit.client.0.smithi155.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.433097+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.290 INFO:tasks.workunit.client.0.smithi155.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovering+degraded+wait 2020-03-11T22:42:50.435300+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.291 INFO:tasks.workunit.client.0.smithi155.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.435766+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.291 INFO:tasks.workunit.client.0.smithi155.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.435644+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.291 INFO:tasks.workunit.client.0.smithi155.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.433024+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.291 INFO:tasks.workunit.client.0.smithi155.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.431959+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.291 INFO:tasks.workunit.client.0.smithi155.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.430877+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.291 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.436793+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.292 INFO:tasks.workunit.client.0.smithi155.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:42:50.436101+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.292 INFO:tasks.workunit.client.0.smithi155.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.436345+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.292 INFO:tasks.workunit.client.0.smithi155.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.382425+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.292 INFO:tasks.workunit.client.0.smithi155.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.431607+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.292 INFO:tasks.workunit.client.0.smithi155.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovery_wait+degraded+wait 2020-03-11T22:42:50.432147+0000 29'100 47:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.292 INFO:tasks.workunit.client.0.smithi155.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.437035+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.292 INFO:tasks.workunit.client.0.smithi155.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.434534+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.293 INFO:tasks.workunit.client.0.smithi155.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:42:50.430624+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.293 INFO:tasks.workunit.client.0.smithi155.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:42:50.432624+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.293 INFO:tasks.workunit.client.0.smithi155.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:42:50.432698+0000 42'101 47:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.293 INFO:tasks.workunit.client.0.smithi155.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.436056+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:03.293 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:43:03.293 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:03.294 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:43:03.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:779: TEST_divergent_3: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:43:03.298 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:03.298 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:03.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:03.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:03.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:03.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:03.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:782: TEST_divergent_3: expfile=td/divergent-priors/exp.17464.out 2020-03-11T22:43:03.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:783: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17464.out 2020-03-11T22:43:03.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:43:03.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:43:03.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:43:03.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:43:03.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:43:03.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op export-remove --pgid 1.0 --file td/divergent-priors/exp.17464.out 2020-03-11T22:43:03.991 INFO:tasks.workunit.client.0.smithi155.stderr:Exporting 1.0 info 1.0( v 29'100 (29'90,29'100] local-lis/les=46/47 n=3 ec=21/21 lis/c=46/46 les/c/f=47/47/0 sis=46 pruub=12.473118472s) 2020-03-11T22:43:04.006 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:023dd3d2:::existing_12:head# 2020-03-11T22:43:04.021 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:04c40a0c:::existing_93:head# 2020-03-11T22:43:04.037 INFO:tasks.workunit.client.0.smithi155.stderr:Read #1:064ff5fb:::existing_36:head# 2020-03-11T22:43:04.046 INFO:tasks.workunit.client.0.smithi155.stderr:Export successful 2020-03-11T22:43:04.047 INFO:tasks.workunit.client.0.smithi155.stdout: marking collection for removal 2020-03-11T22:43:04.047 INFO:tasks.workunit.client.0.smithi155.stdout:setting '_remove' omap key 2020-03-11T22:43:04.047 INFO:tasks.workunit.client.0.smithi155.stdout:finish_remove_pgs 1.0_head removing 1.0 2020-03-11T22:43:04.069 INFO:tasks.workunit.client.0.smithi155.stderr:Remove successful 2020-03-11T22:43:04.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:784: TEST_divergent_3: _objectstore_tool_nodown td/divergent-priors 1 --op import --file td/divergent-priors/exp.17464.out 2020-03-11T22:43:04.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:43:04.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:43:04.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:43:04.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:43:04.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:43:04.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op import --file td/divergent-priors/exp.17464.out 2020-03-11T22:43:05.032 INFO:tasks.workunit.client.0.smithi155.stdout:get_pg_num_history pg_num_history pg_num_history(e47 pg_nums {1={21=1,41=32}} deleted_pools ) 2020-03-11T22:43:05.032 INFO:tasks.workunit.client.0.smithi155.stdout:Importing pgid 1.0 2020-03-11T22:43:05.032 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:023dd3d2:::existing_12:head# 2020-03-11T22:43:05.033 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:43:05.103 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:04c40a0c:::existing_93:head# 2020-03-11T22:43:05.103 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:43:05.111 INFO:tasks.workunit.client.0.smithi155.stdout:Write #1:064ff5fb:::existing_36:head# 2020-03-11T22:43:05.111 INFO:tasks.workunit.client.0.smithi155.stdout:snapset 0=[]:{} 2020-03-11T22:43:05.119 INFO:tasks.workunit.client.0.smithi155.stdout:write_pg epoch 47 info 1.0( v 29'100 (29'90,29'100] local-lis/les=46/47 n=3 ec=21/21 lis/c=46/46 les/c/f=47/47/0 sis=46 pruub=12.473118472s) 2020-03-11T22:43:05.127 INFO:tasks.workunit.client.0.smithi155.stdout:Import successful 2020-03-11T22:43:05.651 INFO:tasks.workunit.client.0.smithi155.stdout:reviving divergent 1 2020-03-11T22:43:05.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:786: TEST_divergent_3: echo 'reviving divergent 1' 2020-03-11T22:43:05.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:787: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:05.983 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:05.983 INFO:tasks.workunit.client.0.smithi155.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.436618+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.983 INFO:tasks.workunit.client.0.smithi155.stdout:1.1e 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.436347+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.984 INFO:tasks.workunit.client.0.smithi155.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.434596+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.984 INFO:tasks.workunit.client.0.smithi155.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.433720+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.984 INFO:tasks.workunit.client.0.smithi155.stdout:1.1b 3 1 1 0 0 1509 0 0 10 10 active+recovery_wait+degraded+wait 2020-03-11T22:42:50.432702+0000 29'100 47:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.984 INFO:tasks.workunit.client.0.smithi155.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.434284+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.984 INFO:tasks.workunit.client.0.smithi155.stdout:1.19 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.433985+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.984 INFO:tasks.workunit.client.0.smithi155.stdout:1.18 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.434166+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.984 INFO:tasks.workunit.client.0.smithi155.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.433503+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.985 INFO:tasks.workunit.client.0.smithi155.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.432156+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.985 INFO:tasks.workunit.client.0.smithi155.stdout:1.15 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.435547+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.985 INFO:tasks.workunit.client.0.smithi155.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.434516+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.985 INFO:tasks.workunit.client.0.smithi155.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.433817+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.985 INFO:tasks.workunit.client.0.smithi155.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.433097+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.985 INFO:tasks.workunit.client.0.smithi155.stdout:1.11 3 1 1 0 0 1509 0 0 16 16 active+recovering+degraded+wait 2020-03-11T22:42:50.435300+0000 29'100 47:126 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.986 INFO:tasks.workunit.client.0.smithi155.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.435766+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.986 INFO:tasks.workunit.client.0.smithi155.stdout:1.5 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.435644+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.986 INFO:tasks.workunit.client.0.smithi155.stdout:1.4 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.433024+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.986 INFO:tasks.workunit.client.0.smithi155.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.431959+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.986 INFO:tasks.workunit.client.0.smithi155.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.430877+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.986 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.436793+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.987 INFO:tasks.workunit.client.0.smithi155.stdout:1.1 4 0 0 0 0 1510 0 0 15 15 active+clean+wait 2020-03-11T22:42:50.436101+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.987 INFO:tasks.workunit.client.0.smithi155.stdout:1.6 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.436345+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.987 INFO:tasks.workunit.client.0.smithi155.stdout:1.7 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.382425+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.987 INFO:tasks.workunit.client.0.smithi155.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.431607+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.987 INFO:tasks.workunit.client.0.smithi155.stdout:1.9 3 1 1 0 0 1510 0 0 11 11 active+recovery_wait+degraded+wait 2020-03-11T22:42:50.432147+0000 29'100 47:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.987 INFO:tasks.workunit.client.0.smithi155.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.437035+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.987 INFO:tasks.workunit.client.0.smithi155.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean+wait 2020-03-11T22:42:50.434534+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.987 INFO:tasks.workunit.client.0.smithi155.stdout:1.c 3 0 0 0 0 1510 0 0 14 14 active+clean+wait 2020-03-11T22:42:50.430624+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.988 INFO:tasks.workunit.client.0.smithi155.stdout:1.d 3 0 0 0 0 1509 0 0 12 12 active+clean+wait 2020-03-11T22:42:50.432624+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.988 INFO:tasks.workunit.client.0.smithi155.stdout:1.e 3 1 1 0 0 1156 0 0 14 14 active+recovery_wait+degraded+wait 2020-03-11T22:42:50.432698+0000 42'101 47:125 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.988 INFO:tasks.workunit.client.0.smithi155.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean+wait 2020-03-11T22:42:50.436056+0000 29'100 47:127 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:05.988 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:43:05.988 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:05.989 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:43:05.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:788: TEST_divergent_3: activate_osd td/divergent-priors 1 2020-03-11T22:43:05.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:43:05.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:43:05.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:43:05.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:43:05.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:43:05.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:05.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:05.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:05.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:05.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:43:05.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:43:05.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:05.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:43:05.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:05.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:43:05.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:05.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:05.999 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:05.999 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:05.999 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:43:05.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:43:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:43:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:06.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:43:06.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:43:06.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:43:06.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:43:06.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:06.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:43:06.001 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:43:06.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:43:06.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:43:06.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:43:06.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:43:06.022 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:06.020+0000 7fbc11fa0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:06.024 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:06.022+0000 7fbc11fa0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:06.025 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:06.022+0000 7fbc11fa0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:06.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:43:06.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:06.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:43:06.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:06.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:06.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:06.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:06.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:06.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:43:06.347 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:43:06.540 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:06.538+0000 7fbc11fa0ec0 -1 Falling back to public interface 2020-03-11T22:43:06.694 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [9,29) [v2:127.0.0.1:6818/42118,v1:127.0.0.1:6819/42118] [v2:127.0.0.1:6820/42118,v1:127.0.0.1:6821/42118] exists,up ce1120e9-d62c-42de-b0b2-a04b6602228c 2020-03-11T22:43:06.694 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:43:06.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:06.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:06.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:06.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:789: TEST_divergent_3: wait_for_osd up 1 2020-03-11T22:43:06.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:06.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:43:06.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:06.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:06.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:06.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:06.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:06.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:43:07.059 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 46 up_thru 46 down_at 31 last_clean_interval [9,29) [v2:127.0.0.1:6818/42118,v1:127.0.0.1:6819/42118] [v2:127.0.0.1:6820/42118,v1:127.0.0.1:6821/42118] exists,up ce1120e9-d62c-42de-b0b2-a04b6602228c 2020-03-11T22:43:07.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:07.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:07.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:07.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:791: TEST_divergent_3: sleep 20 2020-03-11T22:43:07.372 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:07.369+0000 7fbc11fa0ec0 -1 osd.1 47 log_to_monitors {default=true} 2020-03-11T22:43:27.061 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: get_asok_path osd.1 2020-03-11T22:43:27.061 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:43:27.061 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:43:27.062 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:27.062 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:27.062 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:43:27.062 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.1.asok 2020-03-11T22:43:27.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: CEPH_ARGS= 2020-03-11T22:43:27.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:792: TEST_divergent_3: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.1.asok dump_ops_in_flight 2020-03-11T22:43:27.161 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:43:27.161 INFO:tasks.workunit.client.0.smithi155.stdout: "ops": [], 2020-03-11T22:43:27.161 INFO:tasks.workunit.client.0.smithi155.stdout: "num_ops": 0 2020-03-11T22:43:27.162 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:43:27.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:794: TEST_divergent_3: echo 'allowing recovery' 2020-03-11T22:43:27.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:795: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:27.169 INFO:tasks.workunit.client.0.smithi155.stdout:allowing recovery 2020-03-11T22:43:27.493 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:27.493 INFO:tasks.workunit.client.0.smithi155.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:18.939067+0000 29'100 56:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.494 INFO:tasks.workunit.client.0.smithi155.stdout:1.1e 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:43:18.569318+0000 29'100 56:151 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:18.569262+0000 29'100 2020-03-11T22:43:18.569262+0000 0 2020-03-11T22:43:27.494 INFO:tasks.workunit.client.0.smithi155.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:14.914181+0000 29'100 59:132 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.494 INFO:tasks.workunit.client.0.smithi155.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:08.152769+0000 29'100 56:144 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.494 INFO:tasks.workunit.client.0.smithi155.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:18.939174+0000 29'100 56:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.494 INFO:tasks.workunit.client.0.smithi155.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.912098+0000 29'100 56:148 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:11.283538+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.494 INFO:tasks.workunit.client.0.smithi155.stdout:1.19 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:43:22.397001+0000 29'100 59:135 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:22.396956+0000 29'100 2020-03-11T22:43:22.396956+0000 0 2020-03-11T22:43:27.494 INFO:tasks.workunit.client.0.smithi155.stdout:1.18 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:43:19.228530+0000 29'100 56:148 [1,0,2] 1 [1,0,2] 1 29'100 2020-03-11T22:43:19.228469+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.495 INFO:tasks.workunit.client.0.smithi155.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:18.939051+0000 29'100 56:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.495 INFO:tasks.workunit.client.0.smithi155.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.912213+0000 29'100 56:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.495 INFO:tasks.workunit.client.0.smithi155.stdout:1.15 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:43:19.274984+0000 29'100 59:135 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:19.274923+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.495 INFO:tasks.workunit.client.0.smithi155.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:08.150329+0000 29'100 56:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.495 INFO:tasks.workunit.client.0.smithi155.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:18.938049+0000 29'100 56:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.495 INFO:tasks.workunit.client.0.smithi155.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.912323+0000 29'100 56:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.495 INFO:tasks.workunit.client.0.smithi155.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:14.913392+0000 29'100 59:132 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.496 INFO:tasks.workunit.client.0.smithi155.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:08.149953+0000 29'100 56:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.496 INFO:tasks.workunit.client.0.smithi155.stdout:1.5 6 0 0 0 0 2898 0 0 16 16 active+clean 2020-03-11T22:43:20.309693+0000 29'100 59:135 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:20.309641+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.496 INFO:tasks.workunit.client.0.smithi155.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:43:13.286098+0000 29'100 56:147 [1,0,2] 1 [1,0,2] 1 29'100 2020-03-11T22:43:13.286051+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.496 INFO:tasks.workunit.client.0.smithi155.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:43:18.929423+0000 29'100 56:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.496 INFO:tasks.workunit.client.0.smithi155.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:16.911987+0000 29'100 56:148 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:12.318950+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.496 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:08.151856+0000 29'100 56:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.496 INFO:tasks.workunit.client.0.smithi155.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:14.914030+0000 29'100 59:132 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.497 INFO:tasks.workunit.client.0.smithi155.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:43:16.912388+0000 47'101 56:144 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.497 INFO:tasks.workunit.client.0.smithi155.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:43:18.938828+0000 29'100 56:152 [1,2,0] 1 [1,2,0] 1 29'100 2020-03-11T22:43:10.259216+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.497 INFO:tasks.workunit.client.0.smithi155.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:43:08.151254+0000 29'100 56:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.497 INFO:tasks.workunit.client.0.smithi155.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:43:16.303581+0000 29'100 59:135 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:16.303533+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.497 INFO:tasks.workunit.client.0.smithi155.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:16.911753+0000 29'100 56:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.497 INFO:tasks.workunit.client.0.smithi155.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:43:18.938632+0000 29'100 56:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.498 INFO:tasks.workunit.client.0.smithi155.stdout:1.c 3 0 0 0 0 1449 0 0 14 14 active+clean 2020-03-11T22:43:17.255422+0000 29'100 56:148 [1,0,2] 1 [1,0,2] 1 29'100 2020-03-11T22:43:17.255377+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.498 INFO:tasks.workunit.client.0.smithi155.stdout:1.d 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:43:21.349817+0000 29'100 59:135 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:21.349757+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.498 INFO:tasks.workunit.client.0.smithi155.stdout:1.e 3 0 0 0 0 1156 0 0 17 17 active+clean 2020-03-11T22:43:16.911897+0000 42'101 56:149 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.498 INFO:tasks.workunit.client.0.smithi155.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:18.938495+0000 29'100 56:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:27.498 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:43:27.498 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:27.499 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:43:27.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:43:27.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:43:27.626 INFO:tasks.workunit.client.0.smithi155.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:43:27.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:43:27.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:43:27.754 INFO:tasks.workunit.client.0.smithi155.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:43:27.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:797: TEST_divergent_3: for i in $osds 2020-03-11T22:43:27.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:799: TEST_divergent_3: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:43:27.883 INFO:tasks.workunit.client.0.smithi155.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:43:27.891 INFO:tasks.workunit.client.0.smithi155.stdout:reading divergent objects 2020-03-11T22:43:27.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:802: TEST_divergent_3: echo 'reading divergent objects' 2020-03-11T22:43:27.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:803: TEST_divergent_3: ceph pg dump pgs 2020-03-11T22:43:28.212 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:43:28.213 INFO:tasks.workunit.client.0.smithi155.stdout:1.1f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:18.939067+0000 29'100 56:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.213 INFO:tasks.workunit.client.0.smithi155.stdout:1.1e 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:43:18.569318+0000 29'100 56:151 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:18.569262+0000 29'100 2020-03-11T22:43:18.569262+0000 0 2020-03-11T22:43:28.213 INFO:tasks.workunit.client.0.smithi155.stdout:1.1d 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:14.914181+0000 29'100 59:132 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.213 INFO:tasks.workunit.client.0.smithi155.stdout:1.1c 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:08.152769+0000 29'100 56:144 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.213 INFO:tasks.workunit.client.0.smithi155.stdout:1.1b 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:18.939174+0000 29'100 56:153 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.214 INFO:tasks.workunit.client.0.smithi155.stdout:1.1a 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.912098+0000 29'100 56:148 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:11.283538+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.214 INFO:tasks.workunit.client.0.smithi155.stdout:1.19 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:43:22.397001+0000 29'100 59:135 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:22.396956+0000 29'100 2020-03-11T22:43:22.396956+0000 0 2020-03-11T22:43:28.214 INFO:tasks.workunit.client.0.smithi155.stdout:1.18 3 0 0 0 0 1449 0 0 10 10 active+clean 2020-03-11T22:43:19.228530+0000 29'100 56:148 [1,0,2] 1 [1,0,2] 1 29'100 2020-03-11T22:43:19.228469+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.214 INFO:tasks.workunit.client.0.smithi155.stdout:1.17 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:18.939051+0000 29'100 56:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.214 INFO:tasks.workunit.client.0.smithi155.stdout:1.16 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.912213+0000 29'100 56:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.214 INFO:tasks.workunit.client.0.smithi155.stdout:1.15 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:43:19.274984+0000 29'100 59:135 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:19.274923+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.214 INFO:tasks.workunit.client.0.smithi155.stdout:1.14 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:08.150329+0000 29'100 56:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.215 INFO:tasks.workunit.client.0.smithi155.stdout:1.13 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:18.938049+0000 29'100 56:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.215 INFO:tasks.workunit.client.0.smithi155.stdout:1.12 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:16.912323+0000 29'100 56:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.215 INFO:tasks.workunit.client.0.smithi155.stdout:1.11 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:14.913392+0000 29'100 59:132 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.215 INFO:tasks.workunit.client.0.smithi155.stdout:1.10 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:08.149953+0000 29'100 56:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.215 INFO:tasks.workunit.client.0.smithi155.stdout:1.5 6 0 0 0 0 2898 0 0 16 16 active+clean 2020-03-11T22:43:20.309693+0000 29'100 59:135 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:20.309641+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.215 INFO:tasks.workunit.client.0.smithi155.stdout:1.4 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:43:13.286098+0000 29'100 56:147 [1,0,2] 1 [1,0,2] 1 29'100 2020-03-11T22:43:13.286051+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.215 INFO:tasks.workunit.client.0.smithi155.stdout:1.3 4 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:43:18.929423+0000 29'100 56:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.216 INFO:tasks.workunit.client.0.smithi155.stdout:1.2 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:16.911987+0000 29'100 56:148 [0,1,2] 0 [0,1,2] 0 29'100 2020-03-11T22:43:12.318950+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.216 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 3 0 0 0 0 1509 0 0 10 10 active+clean 2020-03-11T22:43:08.151856+0000 29'100 56:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.216 INFO:tasks.workunit.client.0.smithi155.stdout:1.1 4 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:14.914030+0000 29'100 59:132 [2,0,1] 2 [2,0,1] 2 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.216 INFO:tasks.workunit.client.0.smithi155.stdout:1.6 4 0 0 0 0 1993 0 0 17 17 active+clean 2020-03-11T22:43:16.912388+0000 47'101 56:144 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.216 INFO:tasks.workunit.client.0.smithi155.stdout:1.7 2 0 0 0 0 966 0 0 16 16 active+clean 2020-03-11T22:43:18.938828+0000 29'100 56:152 [1,2,0] 1 [1,2,0] 1 29'100 2020-03-11T22:43:10.259216+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.217 INFO:tasks.workunit.client.0.smithi155.stdout:1.8 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:43:08.151254+0000 29'100 56:145 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.217 INFO:tasks.workunit.client.0.smithi155.stdout:1.9 5 0 0 0 0 2415 0 0 16 16 active+clean 2020-03-11T22:43:16.303581+0000 29'100 59:135 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:16.303533+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.217 INFO:tasks.workunit.client.0.smithi155.stdout:1.a 3 0 0 0 0 1510 0 0 16 16 active+clean 2020-03-11T22:43:16.911753+0000 29'100 56:148 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.217 INFO:tasks.workunit.client.0.smithi155.stdout:1.b 3 0 0 0 0 1510 0 0 10 10 active+clean 2020-03-11T22:43:18.938632+0000 29'100 56:151 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.217 INFO:tasks.workunit.client.0.smithi155.stdout:1.c 3 0 0 0 0 1449 0 0 14 14 active+clean 2020-03-11T22:43:17.255422+0000 29'100 56:148 [1,0,2] 1 [1,0,2] 1 29'100 2020-03-11T22:43:17.255377+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.217 INFO:tasks.workunit.client.0.smithi155.stdout:1.d 3 0 0 0 0 1449 0 0 16 16 active+clean 2020-03-11T22:43:21.349817+0000 29'100 59:135 [2,0,1] 2 [2,0,1] 2 29'100 2020-03-11T22:43:21.349757+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.218 INFO:tasks.workunit.client.0.smithi155.stdout:1.e 3 0 0 0 0 1156 0 0 17 17 active+clean 2020-03-11T22:43:16.911897+0000 42'101 56:149 [0,1,2] 0 [0,1,2] 0 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.218 INFO:tasks.workunit.client.0.smithi155.stdout:1.f 3 0 0 0 0 1509 0 0 16 16 active+clean 2020-03-11T22:43:18.938495+0000 29'100 56:149 [1,2,0] 1 [1,2,0] 1 0'0 2020-03-11T22:40:49.940726+0000 0'0 2020-03-11T22:40:49.940726+0000 0 2020-03-11T22:43:28.218 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:43:28.218 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:43:28.219 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:43:28.223 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: expr 2 + 2 2020-03-11T22:43:28.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: seq 1 4 2020-03-11T22:43:28.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:43:28.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:43:28.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:43:28.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:43:28.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:43:28.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:43:28.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:804: TEST_divergent_3: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:43:28.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:806: TEST_divergent_3: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:43:28.342 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: seq 1 2 2020-03-11T22:43:28.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:43:28.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_1 td/divergent-priors/existing 2020-03-11T22:43:28.399 INFO:tasks.workunit.client.0.smithi155.stderr:error getting test/newobject_1: (2) No such file or directory 2020-03-11T22:43:28.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:808: TEST_divergent_3: for i in $(seq 1 $DIVERGENT_CREATE) 2020-03-11T22:43:28.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:810: TEST_divergent_3: rados -p test get newobject_2 td/divergent-priors/existing 2020-03-11T22:43:28.427 INFO:tasks.workunit.client.0.smithi155.stderr:error getting test/newobject_2: (2) No such file or directory 2020-03-11T22:43:28.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:812: TEST_divergent_3: rm -f td/divergent-priors/existing 2020-03-11T22:43:28.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:43:28.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:814: TEST_divergent_3: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:43:28.439 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.066+0000 7fb23de64700 20 _merge_object_divergent_entries: merging hoid 1:97db2a0f:::existing_2:head entries: 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4531.0:1 2020-03-11T22:41:13.427238+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:28.439 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.066+0000 7fb23de64700 20 _merge_object_divergent_entries: keeping 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4531.0:1 2020-03-11T22:41:13.427238+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:28.439 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.066+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:28.440 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.066+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version: 29'2 first_divergent_update: 29'102 last_divergent_update: 29'102 2020-03-11T22:43:28.440 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.066+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head has no more recent entries in log 2020-03-11T22:43:28.440 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.066+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:43:28.440 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.066+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head olog_can_rollback_to: 29'106 2020-03-11T22:43:28.440 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.066+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot rollback 29'102 (29'2) modify 1:97db2a0f:::existing_2:head by client.4531.0:1 2020-03-11T22:41:13.427238+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:28.441 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.066+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head cannot roll back, removing and adding to missing 2020-03-11T22:43:28.441 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.066+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid 1:97db2a0f:::existing_2:head prior_version 29'2 <= info.log_tail 29'89 2020-03-11T22:43:28.441 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.117+0000 7fb23de64700 20 _merge_object_divergent_entries: merging hoid 1:70b3bf12:::existing_4:head entries: 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4543.0:1 2020-03-11T22:41:13.442964+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:28.441 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.117+0000 7fb23de64700 20 _merge_object_divergent_entries: keeping 29'106 (29'4) delete 1:70b3bf12:::existing_4:head by client.4543.0:1 2020-03-11T22:41:13.442964+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:28.441 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.117+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:28.442 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.117+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid 1:70b3bf12:::existing_4:head prior_version: 29'4 first_divergent_update: 29'106 last_divergent_update: 29'106 2020-03-11T22:43:28.442 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.117+0000 7fb23de64700 10 _merge_object_divergent_entries: more recent entry found: 42'101 (29'4) modify 1:70b3bf12:::existing_4:head by client.4654.0:1 2020-03-11T22:42:07.136190+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0, already merged 2020-03-11T22:43:28.442 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.117+0000 7fb23de64700 20 _merge_object_divergent_entries: merging hoid 1:75ac5091:::newobject_2:head entries: 29'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4537.0:1 2020-03-11T22:41:13.438847+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:43:28.442 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.117+0000 7fb23de64700 20 _merge_object_divergent_entries: keeping 29'104 (0'0) modify 1:75ac5091:::newobject_2:head by client.4537.0:1 2020-03-11T22:41:13.438847+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:43:28.442 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.117+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:28.442 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.117+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version: 0'0 first_divergent_update: 29'104 last_divergent_update: 29'104 2020-03-11T22:43:28.443 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.117+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head has no more recent entries in log 2020-03-11T22:43:28.443 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.117+0000 7fb23de64700 10 _merge_object_divergent_entries: hoid 1:75ac5091:::newobject_2:head prior_version or op type indicates creation, deleting 2020-03-11T22:43:28.443 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.118+0000 7fb23ee66700 20 _merge_object_divergent_entries: merging hoid 1:8881e6d8:::existing_3:head entries: 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:41:13.442897+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:28.443 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.118+0000 7fb23ee66700 20 _merge_object_divergent_entries: keeping 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:41:13.442897+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:28.443 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.118+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid object_not_in_store: 1 2020-03-11T22:43:28.443 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.118+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version: 29'3 first_divergent_update: 29'105 last_divergent_update: 29'105 2020-03-11T22:43:28.444 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.118+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head has no more recent entries in log 2020-03-11T22:43:28.444 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.118+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:43:28.444 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.118+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head olog_can_rollback_to: 29'106 2020-03-11T22:43:28.444 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.118+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot rollback 29'105 (29'3) delete 1:8881e6d8:::existing_3:head by client.4542.0:1 2020-03-11T22:41:13.442897+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 0, new_object: 0 2020-03-11T22:43:28.444 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.118+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head cannot roll back, removing and adding to missing 2020-03-11T22:43:28.444 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.118+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:8881e6d8:::existing_3:head prior_version 29'3 <= info.log_tail 29'84 2020-03-11T22:43:28.445 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 20 _merge_object_divergent_entries: merging hoid 1:d858c035:::existing_1:head entries: 29'101 (29'1) modify 1:d858c035:::existing_1:head by client.4530.0:1 2020-03-11T22:41:13.426945+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:28.445 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 20 _merge_object_divergent_entries: keeping 29'101 (29'1) modify 1:d858c035:::existing_1:head by client.4530.0:1 2020-03-11T22:41:13.426945+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:28.445 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:28.445 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version: 29'1 first_divergent_update: 29'101 last_divergent_update: 29'101 2020-03-11T22:43:28.446 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head has no more recent entries in log 2020-03-11T22:43:28.446 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:43:28.446 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head olog_can_rollback_to: 29'106 2020-03-11T22:43:28.446 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot rollback 29'101 (29'1) modify 1:d858c035:::existing_1:head by client.4530.0:1 2020-03-11T22:41:13.426945+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:43:28.446 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head cannot roll back, removing and adding to missing 2020-03-11T22:43:28.447 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:d858c035:::existing_1:head prior_version 29'1 <= info.log_tail 29'90 2020-03-11T22:43:28.447 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 20 _merge_object_divergent_entries: merging hoid 1:da59f6e7:::newobject_1:head entries: 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4536.0:1 2020-03-11T22:41:13.437044+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:43:28.447 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 20 _merge_object_divergent_entries: keeping 29'103 (0'0) modify 1:da59f6e7:::newobject_1:head by client.4536.0:1 2020-03-11T22:41:13.437044+0000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:43:28.447 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:43:28.447 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version: 0'0 first_divergent_update: 29'103 last_divergent_update: 29'103 2020-03-11T22:43:28.448 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head has no more recent entries in log 2020-03-11T22:43:28.448 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:42:50.119+0000 7fb23ee66700 10 _merge_object_divergent_entries: hoid 1:da59f6e7:::newobject_1:head prior_version or op type indicates creation, deleting 2020-03-11T22:43:28.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:43:28.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:816: TEST_divergent_3: grep -q '_merge_object_divergent_entries: more recent entry found:' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:43:28.454 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: find td/divergent-priors -name '*osd*log' 2020-03-11T22:43:28.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:822: TEST_divergent_3: grep -q '_merge_object_divergent_entries.*prior_version or op type indicates creation' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:43:28.462 INFO:tasks.workunit.client.0.smithi155.stdout:success 2020-03-11T22:43:28.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:827: TEST_divergent_3: echo success 2020-03-11T22:43:28.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:829: TEST_divergent_3: rm td/divergent-priors/td/divergent-priors/exp.17464.out 2020-03-11T22:43:28.464 INFO:tasks.workunit.client.0.smithi155.stderr:rm: cannot remove 'td/divergent-priors/td/divergent-priors/exp.17464.out': No such file or directory 2020-03-11T22:43:28.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:831: TEST_divergent_3: delete_pool test 2020-03-11T22:43:28.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:43:28.465 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:43:29.157 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test' does not exist 2020-03-11T22:43:29.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:832: TEST_divergent_3: kill_daemons td/divergent-priors 2020-03-11T22:43:29.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:29.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:29.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:29.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:29.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:29.274 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:29.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:43:29.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:43:29.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:43:29.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:43:29.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:29.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:29.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:29.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:29.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:29.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:29.279 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:43:29.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:43:29.280 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:43:29.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:43:29.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:43:29.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:43:29.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:43:29.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:43:29.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:43:29.284 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:43:29.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:43:29.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:43:29.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:43:29.302 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:43:29.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:29.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:43:29.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17464 2020-03-11T22:43:29.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:43:29.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:43:29.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:44: run: for func in $funcs 2020-03-11T22:43:29.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:45: run: setup td/divergent-priors 2020-03-11T22:43:29.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/divergent-priors 2020-03-11T22:43:29.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/divergent-priors 2020-03-11T22:43:29.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:43:29.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:43:29.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:43:29.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:43:29.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:43:29.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:43:29.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:43:29.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:43:29.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:43:29.308 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:43:29.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:43:29.309 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:43:29.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:43:29.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:43:29.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:43:29.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:43:29.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:43:29.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:43:29.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:43:29.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:43:29.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:43:29.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:43:29.313 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:43:29.313 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:29.314 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:43:29.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17464 2020-03-11T22:43:29.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:43:29.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:43:29.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/divergent-priors 2020-03-11T22:43:29.315 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:43:29.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:29.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:43:29.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.17464 2020-03-11T22:43:29.318 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:43:29.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:43:29.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:43:29.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/divergent-priors 1' TERM HUP INT 2020-03-11T22:43:29.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:46: run: TEST_divergent_ec td/divergent-priors 2020-03-11T22:43:29.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:228: TEST_divergent_ec: local dir=td/divergent-priors 2020-03-11T22:43:29.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:231: TEST_divergent_ec: local dummyfile=/etc/fstab 2020-03-11T22:43:29.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:232: TEST_divergent_ec: local dummyfile2=/etc/resolv.conf 2020-03-11T22:43:29.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:234: TEST_divergent_ec: local num_osds=3 2020-03-11T22:43:29.320 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: expr 3 - 1 2020-03-11T22:43:29.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: seq 0 2 2020-03-11T22:43:29.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:235: TEST_divergent_ec: local 'osds=0 2020-03-11T22:43:29.321 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:43:29.322 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:43:29.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:236: TEST_divergent_ec: run_mon td/divergent-priors a 2020-03-11T22:43:29.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/divergent-priors 2020-03-11T22:43:29.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:43:29.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:43:29.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:43:29.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/divergent-priors/a 2020-03-11T22:43:29.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/divergent-priors/a --run-dir=td/divergent-priors 2020-03-11T22:43:29.655 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:43:29.655 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:29.655 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:29.656 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:29.656 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:29.656 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:43:29.656 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:43:29.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/divergent-priors/a '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --mon-cluster-log-file=td/divergent-priors/log --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:43:29.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:43:29.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:43:29.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:43:29.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:43:29.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:43:29.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:43:29.825 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:43:29.825 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:43:29.825 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:43:29.826 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:29.826 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:29.827 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:43:29.827 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-mon.a.asok 2020-03-11T22:43:29.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:43:29.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17464/ceph-mon.a.asok config get fsid 2020-03-11T22:43:29.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:43:29.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:43:29.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:43:29.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:43:29.913 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:43:29.913 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:43:29.913 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:43:29.913 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:43:29.913 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:43:29.913 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:29.913 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:43:29.914 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-mon.a.asok 2020-03-11T22:43:29.914 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:43:29.914 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.17464/ceph-mon.a.asok config get mon_host 2020-03-11T22:43:29.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:237: TEST_divergent_ec: run_mgr td/divergent-priors x 2020-03-11T22:43:29.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/divergent-priors 2020-03-11T22:43:29.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:43:30.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:43:30.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:43:30.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/divergent-priors/x 2020-03-11T22:43:30.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:43:30.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:43:30.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:30.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:30.907 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:30.907 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:30.907 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:43:30.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:43:30.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:43:30.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/divergent-priors/x '--log-file=td/divergent-priors/$name.log' '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --run-dir=td/divergent-priors '--pid-file=td/divergent-priors/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:43:30.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:30.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 0 2020-03-11T22:43:30.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:43:30.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:43:30.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:43:30.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:43:30.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:43:30.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:30.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:30.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:30.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:30.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:43:30.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:43:30.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:30.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:43:30.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:30.938 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:43:30.938 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:30.939 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:30.939 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:30.939 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:30.939 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:43:30.939 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:43:30.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:43:30.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:30.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:43:30.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:43:30.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:30.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:30.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:30.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:30.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:30.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:43:30.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:43:30.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:43:30.942 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:43:30.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e0e5724e-7440-465c-9db6-82e8c737bcc8 2020-03-11T22:43:30.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 e0e5724e-7440-465c-9db6-82e8c737bcc8' 2020-03-11T22:43:30.942 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:43:30.942 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 e0e5724e-7440-465c-9db6-82e8c737bcc8 2020-03-11T22:43:30.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCSaWleI46GORAAsTWoZMGUYNVRNbAbI8u5qQ== 2020-03-11T22:43:30.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCSaWleI46GORAAsTWoZMGUYNVRNbAbI8u5qQ=="}' 2020-03-11T22:43:30.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e0e5724e-7440-465c-9db6-82e8c737bcc8 -i td/divergent-priors/0/new.json 2020-03-11T22:43:31.295 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:43:31.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/0/new.json 2020-03-11T22:43:31.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCSaWleI46GORAAsTWoZMGUYNVRNbAbI8u5qQ== --osd-uuid e0e5724e-7440-465c-9db6-82e8c737bcc8 2020-03-11T22:43:31.334 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:31.330+0000 7fe403243ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:31.335 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:31.332+0000 7fe403243ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:31.337 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:31.334+0000 7fe403243ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:31.338 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:31.334+0000 7fe403243ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:43:31.338 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:31.334+0000 7fe403243ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:43:31.339 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:31.334+0000 7fe403243ec0 -1 bluestore(td/divergent-priors/0/block) _read_bdev_label failed to open td/divergent-priors/0/block: (2) No such file or directory 2020-03-11T22:43:31.418 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:31.414+0000 7fe403243ec0 -1 bluestore(td/divergent-priors/0) _read_fsid unparsable uuid 2020-03-11T22:43:34.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/0/keyring 2020-03-11T22:43:34.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:43:34.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:43:34.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:43:34.474 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2020-03-11T22:43:34.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:43:34.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:34.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:43:34.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:43:34.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:43:34.951 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:43:34.977 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:34.974+0000 7f86b8e45ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:34.978 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:34.976+0000 7f86b8e45ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:34.980 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:34.978+0000 7f86b8e45ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:35.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:43:35.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:35.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:43:35.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:35.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:35.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:35.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:35.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:35.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:35.298 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:43:35.496 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:35.493+0000 7f86b8e45ec0 -1 Falling back to public interface 2020-03-11T22:43:35.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:36.275 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:36.272+0000 7f86b8e45ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:43:36.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:36.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:36.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:36.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:36.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:43:36.640 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:43:37.011 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/43615,v1:127.0.0.1:6803/43615] [v2:127.0.0.1:6804/43615,v1:127.0.0.1:6805/43615] exists,up e0e5724e-7440-465c-9db6-82e8c737bcc8 2020-03-11T22:43:37.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:37.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:37.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:37.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:37.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 1 2020-03-11T22:43:37.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:43:37.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:43:37.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:43:37.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:43:37.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:43:37.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:37.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:37.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:37.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:37.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:43:37.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:43:37.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:37.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:43:37.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:37.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:43:37.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:37.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:37.014 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:37.014 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:37.014 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:43:37.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:43:37.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:43:37.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:37.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:43:37.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:43:37.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:37.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:37.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:37.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:37.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:37.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:43:37.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:43:37.016 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:43:37.016 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:43:37.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=7b6947d8-603c-41a4-abcb-8b50325a2007 2020-03-11T22:43:37.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 7b6947d8-603c-41a4-abcb-8b50325a2007' 2020-03-11T22:43:37.017 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:43:37.018 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 7b6947d8-603c-41a4-abcb-8b50325a2007 2020-03-11T22:43:37.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCZaWle4A2gARAAaBEhIANx1VqxklukyWSDng== 2020-03-11T22:43:37.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCZaWle4A2gARAAaBEhIANx1VqxklukyWSDng=="}' 2020-03-11T22:43:37.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 7b6947d8-603c-41a4-abcb-8b50325a2007 -i td/divergent-priors/1/new.json 2020-03-11T22:43:37.550 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:43:37.560 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/1/new.json 2020-03-11T22:43:37.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCZaWle4A2gARAAaBEhIANx1VqxklukyWSDng== --osd-uuid 7b6947d8-603c-41a4-abcb-8b50325a2007 2020-03-11T22:43:37.589 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:37.586+0000 7fa86dee5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:37.611 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:37.609+0000 7fa86dee5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:37.613 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:37.610+0000 7fa86dee5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:37.614 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:37.611+0000 7fa86dee5ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:43:37.614 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:37.611+0000 7fa86dee5ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:43:37.614 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:37.611+0000 7fa86dee5ec0 -1 bluestore(td/divergent-priors/1/block) _read_bdev_label failed to open td/divergent-priors/1/block: (2) No such file or directory 2020-03-11T22:43:37.719 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:37.717+0000 7fa86dee5ec0 -1 bluestore(td/divergent-priors/1) _read_fsid unparsable uuid 2020-03-11T22:43:40.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/1/keyring 2020-03-11T22:43:40.782 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:43:40.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:43:40.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:43:40.783 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2020-03-11T22:43:41.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:43:41.243 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:43:41.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:41.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:43:41.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:43:41.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:43:41.270 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:41.268+0000 7f3a1ee27ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:41.272 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:41.270+0000 7f3a1ee27ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:41.274 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:41.272+0000 7f3a1ee27ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:41.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:43:41.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:41.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:43:41.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:41.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:41.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:41.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:41.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:41.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:43:41.595 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:43:41.791 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:41.788+0000 7f3a1ee27ec0 -1 Falling back to public interface 2020-03-11T22:43:41.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:42.593 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:42.590+0000 7f3a1ee27ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:43:42.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:42.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:42.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:42.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:42.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:43:42.942 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:43:43.309 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/43970,v1:127.0.0.1:6811/43970] [v2:127.0.0.1:6812/43970,v1:127.0.0.1:6813/43970] exists,up 7b6947d8-603c-41a4-abcb-8b50325a2007 2020-03-11T22:43:43.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:43.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:43.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:43.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:238: TEST_divergent_ec: for i in $osds 2020-03-11T22:43:43.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:240: TEST_divergent_ec: run_osd td/divergent-priors 2 2020-03-11T22:43:43.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/divergent-priors 2020-03-11T22:43:43.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:43:43.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:43:43.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:43:43.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:43:43.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:43:43.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:43:43.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:43:43.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:43:43.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:43:43.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:43:43.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:43:43.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:43:43.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:43:43.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:43:43.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:43:43.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:43:43.312 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:43:43.312 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:43:43.312 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:43:43.313 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:43:43.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:43:43.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:43:43.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:43:43.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:43:43.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:43:43.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:43:43.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:43:43.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:43:43.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:43.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:43:43.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:43:43.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:43:43.314 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:43:43.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=797c47f4-417e-4663-8ca9-e4cc78330dd9 2020-03-11T22:43:43.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 797c47f4-417e-4663-8ca9-e4cc78330dd9' 2020-03-11T22:43:43.315 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:43:43.316 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 797c47f4-417e-4663-8ca9-e4cc78330dd9 2020-03-11T22:43:43.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCfaWlefOdhExAAUZAYrEbZPu0GbUiw7pSMvw== 2020-03-11T22:43:43.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCfaWlefOdhExAAUZAYrEbZPu0GbUiw7pSMvw=="}' 2020-03-11T22:43:43.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 797c47f4-417e-4663-8ca9-e4cc78330dd9 -i td/divergent-priors/2/new.json 2020-03-11T22:43:43.850 INFO:tasks.workunit.client.0.smithi155.stdout:2 2020-03-11T22:43:43.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/divergent-priors/2/new.json 2020-03-11T22:43:43.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCfaWlefOdhExAAUZAYrEbZPu0GbUiw7pSMvw== --osd-uuid 797c47f4-417e-4663-8ca9-e4cc78330dd9 2020-03-11T22:43:43.890 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:43.887+0000 7f3692a03ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:43.917 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:43.914+0000 7f3692a03ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:43.919 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:43.917+0000 7f3692a03ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:43.919 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:43.917+0000 7f3692a03ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:43:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:43.917+0000 7f3692a03ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:43:43.920 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:43.917+0000 7f3692a03ec0 -1 bluestore(td/divergent-priors/2/block) _read_bdev_label failed to open td/divergent-priors/2/block: (2) No such file or directory 2020-03-11T22:43:44.017 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:44.015+0000 7f3692a03ec0 -1 bluestore(td/divergent-priors/2) _read_fsid unparsable uuid 2020-03-11T22:43:47.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/divergent-priors/2/keyring 2020-03-11T22:43:47.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:43:47.145 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2020-03-11T22:43:47.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:43:47.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/divergent-priors/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:43:47.601 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:43:47.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:43:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:43:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:43:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:43:47.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:43:47.629 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:47.626+0000 7fa9ad76bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:47.630 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:47.628+0000 7fa9ad76bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:47.632 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:47.629+0000 7fa9ad76bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:43:47.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:43:47.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:43:47.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:43:47.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:43:47.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:43:47.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:47.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:43:47.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:47.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:47.944 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:43:48.147 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:48.145+0000 7fa9ad76bec0 -1 Falling back to public interface 2020-03-11T22:43:48.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:43:48.899 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:43:48.897+0000 7fa9ad76bec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:43:49.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:43:49.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:43:49.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:43:49.293 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:43:49.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:43:49.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:43:49.682 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/44325,v1:127.0.0.1:6819/44325] [v2:127.0.0.1:6820/44325,v1:127.0.0.1:6821/44325] exists,up 797c47f4-417e-4663-8ca9-e4cc78330dd9 2020-03-11T22:43:49.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:43:49.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:43:49.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:43:49.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:243: TEST_divergent_ec: ceph osd set noout 2020-03-11T22:43:50.418 INFO:tasks.workunit.client.0.smithi155.stderr:noout is set 2020-03-11T22:43:50.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:244: TEST_divergent_ec: ceph osd set noin 2020-03-11T22:43:51.136 INFO:tasks.workunit.client.0.smithi155.stderr:noin is set 2020-03-11T22:43:51.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:245: TEST_divergent_ec: ceph osd set nodown 2020-03-11T22:43:51.905 INFO:tasks.workunit.client.0.smithi155.stderr:nodown is set 2020-03-11T22:43:51.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:246: TEST_divergent_ec: create_ec_pool test true k=2 m=1 2020-03-11T22:43:51.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: create_ec_pool: local pool_name=test 2020-03-11T22:43:51.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: create_ec_pool: shift 2020-03-11T22:43:51.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: create_ec_pool: local allow_overwrites=true 2020-03-11T22:43:51.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: create_ec_pool: shift 2020-03-11T22:43:51.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2272: create_ec_pool: ceph osd erasure-code-profile set myprofile crush-failure-domain=osd k=2 m=1 2020-03-11T22:43:52.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2274: create_ec_pool: create_pool test 1 1 erasure myprofile 2020-03-11T22:43:52.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 erasure myprofile 2020-03-11T22:43:53.135 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test' already exists 2020-03-11T22:43:53.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:43:54.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2276: create_ec_pool: '[' true = true ']' 2020-03-11T22:43:54.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2277: create_ec_pool: ceph osd pool set test allow_ec_overwrites true 2020-03-11T22:43:54.853 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 allow_ec_overwrites to true 2020-03-11T22:43:54.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: create_ec_pool: wait_for_clean 2020-03-11T22:43:54.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:43:54.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:43:54.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:43:54.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:43:54.862 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:43:54.862 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:43:54.863 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:43:54.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:43:54.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:43:54.863 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:43:54.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:43:54.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:43:54.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:43:54.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:43:54.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:43:55.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:43:55.318 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:43:55.318 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:43:55.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:43:55.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:55.318 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:43:55.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836487 2020-03-11T22:43:55.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487' 2020-03-11T22:43:55.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:55.449 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:43:55.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705670 2020-03-11T22:43:55.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-38654705670' 2020-03-11T22:43:55.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:55.579 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:43:55.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542148 2020-03-11T22:43:55.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836487 1-38654705670 2-60129542148' 2020-03-11T22:43:55.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:55.709 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836487 2020-03-11T22:43:55.709 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:55.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:43:55.711 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836487 2020-03-11T22:43:55.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:55.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836487 2020-03-11T22:43:55.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836487' 2020-03-11T22:43:55.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:55.713 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836487 2020-03-11T22:43:56.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836485 -lt 21474836487 2020-03-11T22:43:56.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:43:57.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:43:57.055 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:43:57.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836487 -lt 21474836487 2020-03-11T22:43:57.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:57.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705670 2020-03-11T22:43:57.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:57.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:43:57.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705670 2020-03-11T22:43:57.409 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:57.409 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705670 2020-03-11T22:43:57.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705670' 2020-03-11T22:43:57.410 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705670 2020-03-11T22:43:57.410 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:43:57.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705670 -lt 38654705670 2020-03-11T22:43:57.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:43:57.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542148 2020-03-11T22:43:57.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:43:57.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:43:57.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542148 2020-03-11T22:43:57.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:43:57.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542148 2020-03-11T22:43:57.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542148' 2020-03-11T22:43:57.754 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542148 2020-03-11T22:43:57.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:43:58.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542148 -lt 60129542148 2020-03-11T22:43:58.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:43:58.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:58.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:58.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:43:58.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:43:58.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:43:58.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:43:58.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:43:58.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:43:58.545 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:43:58.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:43:58.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:43:58.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:43:58.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:43:58.888 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:43:59.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:43:59.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:43:59.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:43:59.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2281: create_ec_pool: return 0 2020-03-11T22:43:59.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:248: TEST_divergent_ec: flush_pg_stats 2020-03-11T22:43:59.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:43:59.337 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:43:59.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:43:59.680 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:43:59.680 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:43:59.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:43:59.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:59.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:43:59.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:43:59.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:43:59.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:59.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:43:59.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705673 2020-03-11T22:43:59.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-38654705673' 2020-03-11T22:43:59.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:43:59.939 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:00.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542150 2020-03-11T22:44:00.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-38654705673 2-60129542150' 2020-03-11T22:44:00.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:00.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:44:00.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:00.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:00.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:44:00.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:00.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:44:00.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:44:00.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:00.084 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:44:00.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836490 2020-03-11T22:44:00.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:00.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705673 2020-03-11T22:44:00.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:00.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:00.499 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705673 2020-03-11T22:44:00.499 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:00.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705673 2020-03-11T22:44:00.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705673' 2020-03-11T22:44:00.500 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705673 2020-03-11T22:44:00.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:00.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705673 -lt 38654705673 2020-03-11T22:44:00.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:00.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542150 2020-03-11T22:44:00.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:00.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:00.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542150 2020-03-11T22:44:00.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:00.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542150 2020-03-11T22:44:00.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2020-03-11T22:44:00.849 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:00.850 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542150 2020-03-11T22:44:01.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542151 -lt 60129542150 2020-03-11T22:44:01.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:249: TEST_divergent_ec: wait_for_clean 2020-03-11T22:44:01.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:44:01.232 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:44:01.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:44:01.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:44:01.233 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:44:01.233 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:44:01.233 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:44:01.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:44:01.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:44:01.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:44:01.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:44:01.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:44:01.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:44:01.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:44:01.326 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:44:01.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:44:01.666 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:44:01.666 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:44:01.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:44:01.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:01.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:44:01.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836492 2020-03-11T22:44:01.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492' 2020-03-11T22:44:01.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:01.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:44:01.925 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705675 2020-03-11T22:44:01.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-38654705675' 2020-03-11T22:44:01.926 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:01.926 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:02.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542153 2020-03-11T22:44:02.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836492 1-38654705675 2-60129542153' 2020-03-11T22:44:02.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:02.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836492 2020-03-11T22:44:02.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:02.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:02.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836492 2020-03-11T22:44:02.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:02.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836492 2020-03-11T22:44:02.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836492' 2020-03-11T22:44:02.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:02.059 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836492 2020-03-11T22:44:02.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836492 -lt 21474836492 2020-03-11T22:44:02.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:02.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705675 2020-03-11T22:44:02.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:02.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:02.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705675 2020-03-11T22:44:02.417 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:02.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705675 2020-03-11T22:44:02.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705675' 2020-03-11T22:44:02.420 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705675 2020-03-11T22:44:02.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:02.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705675 -lt 38654705675 2020-03-11T22:44:02.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:02.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542153 2020-03-11T22:44:02.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:02.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:02.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542153 2020-03-11T22:44:02.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:02.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542153 2020-03-11T22:44:02.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2020-03-11T22:44:02.766 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542153 2020-03-11T22:44:02.767 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:03.149 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542153 -lt 60129542153 2020-03-11T22:44:03.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:44:03.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:03.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:03.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:44:03.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:03.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:03.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:03.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:03.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:03.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:03.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:03.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:44:03.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:03.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:03.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:04.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:44:04.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:44:04.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:44:04.371 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: ceph pg dump pgs --format=json 2020-03-11T22:44:04.371 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: jq '.pg_stats[0].up_primary' 2020-03-11T22:44:04.691 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:44:04.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:252: TEST_divergent_ec: local divergent=1 2020-03-11T22:44:04.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:253: TEST_divergent_ec: echo 'primary and soon to be divergent is 1' 2020-03-11T22:44:04.700 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:254: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:04.700 INFO:tasks.workunit.client.0.smithi155.stdout:primary and soon to be divergent is 1 2020-03-11T22:44:05.016 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:05.017 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:43:53.851441+0000 0'0 26:12 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:44:05.017 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:44:05.017 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:05.018 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:44:05.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:255: TEST_divergent_ec: local non_divergent= 2020-03-11T22:44:05.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:05.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 0 = 1 ']' 2020-03-11T22:44:05.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0' 2020-03-11T22:44:05.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:05.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 1 = 1 ']' 2020-03-11T22:44:05.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:259: TEST_divergent_ec: continue 2020-03-11T22:44:05.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:256: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:05.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:258: TEST_divergent_ec: '[' 2 = 1 ']' 2020-03-11T22:44:05.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:261: TEST_divergent_ec: non_divergent=' 0 2' 2020-03-11T22:44:05.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:264: TEST_divergent_ec: echo 'writing initial objects' 2020-03-11T22:44:05.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: seq 1 100 2020-03-11T22:44:05.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_1 /etc/fstab 2020-03-11T22:44:05.030 INFO:tasks.workunit.client.0.smithi155.stdout:writing initial objects 2020-03-11T22:44:05.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_2 /etc/fstab 2020-03-11T22:44:05.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_3 /etc/fstab 2020-03-11T22:44:05.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_4 /etc/fstab 2020-03-11T22:44:05.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_5 /etc/fstab 2020-03-11T22:44:05.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_6 /etc/fstab 2020-03-11T22:44:05.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_7 /etc/fstab 2020-03-11T22:44:05.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_8 /etc/fstab 2020-03-11T22:44:05.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_9 /etc/fstab 2020-03-11T22:44:05.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_10 /etc/fstab 2020-03-11T22:44:05.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_11 /etc/fstab 2020-03-11T22:44:05.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_12 /etc/fstab 2020-03-11T22:44:05.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:05.928 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_13 /etc/fstab 2020-03-11T22:44:06.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_14 /etc/fstab 2020-03-11T22:44:06.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_15 /etc/fstab 2020-03-11T22:44:06.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_16 /etc/fstab 2020-03-11T22:44:06.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_17 /etc/fstab 2020-03-11T22:44:06.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_18 /etc/fstab 2020-03-11T22:44:06.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_19 /etc/fstab 2020-03-11T22:44:06.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_20 /etc/fstab 2020-03-11T22:44:06.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_21 /etc/fstab 2020-03-11T22:44:06.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_22 /etc/fstab 2020-03-11T22:44:06.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_23 /etc/fstab 2020-03-11T22:44:06.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_24 /etc/fstab 2020-03-11T22:44:06.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_25 /etc/fstab 2020-03-11T22:44:06.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_26 /etc/fstab 2020-03-11T22:44:06.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:06.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_27 /etc/fstab 2020-03-11T22:44:07.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_28 /etc/fstab 2020-03-11T22:44:07.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_29 /etc/fstab 2020-03-11T22:44:07.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_30 /etc/fstab 2020-03-11T22:44:07.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_31 /etc/fstab 2020-03-11T22:44:07.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.287 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_32 /etc/fstab 2020-03-11T22:44:07.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.329 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_33 /etc/fstab 2020-03-11T22:44:07.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_34 /etc/fstab 2020-03-11T22:44:07.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_35 /etc/fstab 2020-03-11T22:44:07.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_36 /etc/fstab 2020-03-11T22:44:07.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_37 /etc/fstab 2020-03-11T22:44:07.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_38 /etc/fstab 2020-03-11T22:44:07.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.863 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_39 /etc/fstab 2020-03-11T22:44:07.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_40 /etc/fstab 2020-03-11T22:44:07.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_41 /etc/fstab 2020-03-11T22:44:07.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:07.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_42 /etc/fstab 2020-03-11T22:44:08.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_43 /etc/fstab 2020-03-11T22:44:08.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_44 /etc/fstab 2020-03-11T22:44:08.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_45 /etc/fstab 2020-03-11T22:44:08.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_46 /etc/fstab 2020-03-11T22:44:08.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_47 /etc/fstab 2020-03-11T22:44:08.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_48 /etc/fstab 2020-03-11T22:44:08.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_49 /etc/fstab 2020-03-11T22:44:08.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_50 /etc/fstab 2020-03-11T22:44:08.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_51 /etc/fstab 2020-03-11T22:44:08.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_52 /etc/fstab 2020-03-11T22:44:08.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_53 /etc/fstab 2020-03-11T22:44:08.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_54 /etc/fstab 2020-03-11T22:44:08.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_55 /etc/fstab 2020-03-11T22:44:08.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:08.938 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_56 /etc/fstab 2020-03-11T22:44:09.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_57 /etc/fstab 2020-03-11T22:44:09.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.063 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_58 /etc/fstab 2020-03-11T22:44:09.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_59 /etc/fstab 2020-03-11T22:44:09.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_60 /etc/fstab 2020-03-11T22:44:09.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_61 /etc/fstab 2020-03-11T22:44:09.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_62 /etc/fstab 2020-03-11T22:44:09.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_63 /etc/fstab 2020-03-11T22:44:09.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_64 /etc/fstab 2020-03-11T22:44:09.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_65 /etc/fstab 2020-03-11T22:44:09.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_66 /etc/fstab 2020-03-11T22:44:09.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.620 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_67 /etc/fstab 2020-03-11T22:44:09.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_68 /etc/fstab 2020-03-11T22:44:09.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_69 /etc/fstab 2020-03-11T22:44:09.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_70 /etc/fstab 2020-03-11T22:44:09.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_71 /etc/fstab 2020-03-11T22:44:09.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:09.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_72 /etc/fstab 2020-03-11T22:44:10.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_73 /etc/fstab 2020-03-11T22:44:10.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_74 /etc/fstab 2020-03-11T22:44:10.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_75 /etc/fstab 2020-03-11T22:44:10.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_76 /etc/fstab 2020-03-11T22:44:10.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_77 /etc/fstab 2020-03-11T22:44:10.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_78 /etc/fstab 2020-03-11T22:44:10.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_79 /etc/fstab 2020-03-11T22:44:10.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_80 /etc/fstab 2020-03-11T22:44:10.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_81 /etc/fstab 2020-03-11T22:44:10.788 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_82 /etc/fstab 2020-03-11T22:44:10.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_83 /etc/fstab 2020-03-11T22:44:10.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.914 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_84 /etc/fstab 2020-03-11T22:44:10.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:10.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_85 /etc/fstab 2020-03-11T22:44:11.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_86 /etc/fstab 2020-03-11T22:44:11.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_87 /etc/fstab 2020-03-11T22:44:11.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_88 /etc/fstab 2020-03-11T22:44:11.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.272 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_89 /etc/fstab 2020-03-11T22:44:11.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_90 /etc/fstab 2020-03-11T22:44:11.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_91 /etc/fstab 2020-03-11T22:44:11.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_92 /etc/fstab 2020-03-11T22:44:11.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_93 /etc/fstab 2020-03-11T22:44:11.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_94 /etc/fstab 2020-03-11T22:44:11.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_95 /etc/fstab 2020-03-11T22:44:11.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_96 /etc/fstab 2020-03-11T22:44:11.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_97 /etc/fstab 2020-03-11T22:44:11.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_98 /etc/fstab 2020-03-11T22:44:11.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:44:11.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:266: TEST_divergent_ec: for i in $(seq 1 $testobjects) 2020-03-11T22:44:11.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:268: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:44:12.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:44:12.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:271: TEST_divergent_ec: wait_for_clean 2020-03-11T22:44:12.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:44:12.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:44:12.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:44:12.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:44:12.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:44:12.076 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:44:12.076 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:44:12.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:44:12.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:44:12.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:44:12.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:44:12.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:44:12.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:44:12.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:44:12.133 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:44:12.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:44:12.477 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:44:12.478 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:44:12.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:44:12.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:12.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:44:12.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836497 2020-03-11T22:44:12.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836497' 2020-03-11T22:44:12.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:12.648 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:44:12.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705679 2020-03-11T22:44:12.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836497 1-38654705679' 2020-03-11T22:44:12.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:12.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:12.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542157 2020-03-11T22:44:12.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836497 1-38654705679 2-60129542157' 2020-03-11T22:44:12.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:12.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836497 2020-03-11T22:44:12.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:12.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:12.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836497 2020-03-11T22:44:12.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:12.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836497 2020-03-11T22:44:12.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836497' 2020-03-11T22:44:12.909 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:12.910 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836497 2020-03-11T22:44:13.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836494 -lt 21474836497 2020-03-11T22:44:13.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:44:14.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:44:14.250 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:14.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836497 -lt 21474836497 2020-03-11T22:44:14.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:14.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705679 2020-03-11T22:44:14.604 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:14.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:14.605 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705679 2020-03-11T22:44:14.605 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:14.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705679 2020-03-11T22:44:14.606 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705679' 2020-03-11T22:44:14.606 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:14.606 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705679 2020-03-11T22:44:14.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705680 -lt 38654705679 2020-03-11T22:44:14.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:14.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542157 2020-03-11T22:44:14.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:14.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:14.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542157 2020-03-11T22:44:14.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:14.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542157 2020-03-11T22:44:14.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542157' 2020-03-11T22:44:14.950 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:14.951 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542157 2020-03-11T22:44:15.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542157 -lt 60129542157 2020-03-11T22:44:15.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:44:15.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:15.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:15.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:44:15.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:15.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:15.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:15.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:15.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:15.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:15.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:16.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:44:16.161 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:16.161 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:16.161 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:16.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:44:16.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:44:16.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:44:16.611 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: get_pg test existing_1 2020-03-11T22:44:16.611 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: get_pg: local poolname=test 2020-03-11T22:44:16.611 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: get_pg: local objectname=existing_1 2020-03-11T22:44:16.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: ceph --format json osd map test existing_1 2020-03-11T22:44:16.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: get_pg: jq -r .pgid 2020-03-11T22:44:16.958 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:273: TEST_divergent_ec: local pgid=1.0 2020-03-11T22:44:16.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:276: TEST_divergent_ec: echo 'blackholing osds 0 2' 2020-03-11T22:44:16.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:277: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:16.959 INFO:tasks.workunit.client.0.smithi155.stdout:blackholing osds 0 2 2020-03-11T22:44:17.279 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:17.279 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 0 0 0 48300 0 0 16 16 active+clean 2020-03-11T22:43:53.851441+0000 27'100 27:112 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:44:17.280 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:44:17.280 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:17.281 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:44:17.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:17.290 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:44:17.290 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:44:17.290 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:44:17.291 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:17.291 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:17.291 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:44:17.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.0.asok 2020-03-11T22:44:17.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:44:17.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.0.asok config set objectstore_blackhole 1 2020-03-11T22:44:17.377 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:44:17.377 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:44:17.377 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:44:17.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:278: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:17.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:44:17.385 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:44:17.385 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:44:17.385 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:44:17.385 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:17.385 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:44:17.385 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.2.asok 2020-03-11T22:44:17.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:44:17.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:280: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.2.asok config set objectstore_blackhole 1 2020-03-11T22:44:17.471 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:44:17.472 INFO:tasks.workunit.client.0.smithi155.stdout: "success": "objectstore_blackhole = 'true' (not observed, change may require restart) " 2020-03-11T22:44:17.472 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:44:17.478 INFO:tasks.workunit.client.0.smithi155.stdout:writing divergent object 2020-03-11T22:44:17.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:284: TEST_divergent_ec: echo 'writing divergent object' 2020-03-11T22:44:17.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:286: TEST_divergent_ec: sleep 1 2020-03-11T22:44:17.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:285: TEST_divergent_ec: rados -p test put existing_100 /etc/resolv.conf 2020-03-11T22:44:18.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:288: TEST_divergent_ec: rados -p test mksnap snap1 2020-03-11T22:44:18.480 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:287: TEST_divergent_ec: rados -p test put existing_100 /etc/fstab 2020-03-11T22:44:18.645 INFO:tasks.workunit.client.0.smithi155.stdout:created pool test snap snap1 2020-03-11T22:44:18.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:290: TEST_divergent_ec: sleep 10 2020-03-11T22:44:18.647 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: expr 100 - 1 2020-03-11T22:44:18.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:289: TEST_divergent_ec: rados -p test put existing_99 /etc/fstab 2020-03-11T22:44:28.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:291: TEST_divergent_ec: killall -9 rados 2020-03-11T22:44:28.653 INFO:tasks.workunit.client.0.smithi155.stderr:rados: no process found 2020-03-11T22:44:28.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:294: TEST_divergent_ec: echo 'killing all the osds' 2020-03-11T22:44:28.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:295: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:28.653 INFO:tasks.workunit.client.0.smithi155.stdout:killing all the osds 2020-03-11T22:44:28.983 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:28.983 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 active+clean 2020-03-11T22:43:53.851441+0000 28'104 28:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:44:28.983 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:44:28.983 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:28.984 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:44:28.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:296: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd 2020-03-11T22:44:28.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:44:28.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:44:28.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:44:28.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:44:28.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:44:29.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:44:29.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:29.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.0 2020-03-11T22:44:29.632 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already down. 2020-03-11T22:44:29.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:29.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.1 2020-03-11T22:44:30.123 INFO:tasks.workunit.client.0.smithi155.stderr:osd.1 is already down. 2020-03-11T22:44:30.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:297: TEST_divergent_ec: for i in $osds 2020-03-11T22:44:30.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:299: TEST_divergent_ec: ceph osd down osd.2 2020-03-11T22:44:30.657 INFO:tasks.workunit.client.0.smithi155.stderr:osd.2 is already down. 2020-03-11T22:44:30.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:30.665 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.0 2020-03-11T22:44:31.191 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already out. 2020-03-11T22:44:31.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:301: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:31.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:303: TEST_divergent_ec: ceph osd out osd.2 2020-03-11T22:44:31.740 INFO:tasks.workunit.client.0.smithi155.stderr:osd.2 is already out. 2020-03-11T22:44:31.749 INFO:tasks.workunit.client.0.smithi155.stdout:bringing up non_divergent 0 2 2020-03-11T22:44:31.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:307: TEST_divergent_ec: echo 'bringing up non_divergent 0 2' 2020-03-11T22:44:31.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:308: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:32.070 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:32.070 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 stale+active+clean 2020-03-11T22:43:53.851441+0000 28'104 28:116 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:44:32.070 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:44:32.071 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:32.072 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:44:32.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:32.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:44:32.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:32.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:32.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:44:32.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:32.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:44:32.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:32.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:32.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:32.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:32.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:44:32.084 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:44:32.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:32.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:32.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:32.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:32.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:32.085 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:32.085 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:32.086 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:32.086 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:44:32.086 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:44:32.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:44:32.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:32.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:32.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:32.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:32.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:32.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:32.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:32.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:32.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:44:32.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:44:32.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:32.088 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:44:32.088 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:44:32.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:44:32.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:32.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:32.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:32.109 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:44:32.107+0000 7f00f557dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:32.111 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:44:32.109+0000 7f00f557dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:32.113 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:44:32.111+0000 7f00f557dec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:32.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:44:32.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:32.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:44:32.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:32.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:32.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:32.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:32.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:32.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:32.430 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:44:32.629 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:44:32.627+0000 7f00f557dec0 -1 Falling back to public interface 2020-03-11T22:44:32.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:44:33.511 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:44:33.508+0000 7f00f557dec0 -1 osd.0 27 log_to_monitors {default=true} 2020-03-11T22:44:33.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:44:33.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:33.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:44:33.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:33.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:44:33.779 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:44:34.126 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up out weight 0 up_from 34 up_thru 0 down_at 29 last_clean_interval [5,27) [v2:127.0.0.1:6802/48706,v1:127.0.0.1:6803/48706] [v2:127.0.0.1:6804/48706,v1:127.0.0.1:6805/48706] exists,up e0e5724e-7440-465c-9db6-82e8c737bcc8 2020-03-11T22:44:34.126 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:44:34.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:34.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:34.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:34.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:309: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:34.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:311: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:44:34.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:44:34.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:44:34.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:44:34.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:44:34.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:44:34.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:44:34.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:44:34.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:44:34.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:44:34.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:44:34.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:44:34.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:44:34.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:44:34.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:44:34.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:44:34.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:44:34.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:44:34.129 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:44:34.130 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:44:34.130 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:44:34.130 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:44:34.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:44:34.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:44:34.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:44:34.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:44:34.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:44:34.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:44:34.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:34.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:44:34.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:44:34.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:44:34.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:44:34.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:44:34.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:44:34.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:44:34.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:44:34.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:44:34.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:44:34.152 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:44:34.150+0000 7fcd3c2a9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:34.154 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:44:34.152+0000 7fcd3c2a9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:34.156 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:44:34.153+0000 7fcd3c2a9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:44:34.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:44:34.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:44:34.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:44:34.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:44:34.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:44:34.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:34.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:44:34.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:34.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:34.486 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:44:34.678 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:44:34.675+0000 7fcd3c2a9ec0 -1 Falling back to public interface 2020-03-11T22:44:34.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:44:35.562 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:44:35.560+0000 7fcd3c2a9ec0 -1 osd.2 27 log_to_monitors {default=true} 2020-03-11T22:44:35.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:44:35.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:44:35.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:44:35.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:44:35.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:44:35.835 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:44:36.181 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up out weight 0 up_from 35 up_thru 0 down_at 31 last_clean_interval [14,27) [v2:127.0.0.1:6810/48900,v1:127.0.0.1:6811/48900] [v2:127.0.0.1:6812/48900,v1:127.0.0.1:6813/48900] exists,up 797c47f4-417e-4663-8ca9-e4cc78330dd9 2020-03-11T22:44:36.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:44:36.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:44:36.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:44:36.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:36.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.0 2020-03-11T22:44:36.757 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already in. 2020-03-11T22:44:36.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:313: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:44:36.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:315: TEST_divergent_ec: ceph osd in osd.2 2020-03-11T22:44:37.350 INFO:tasks.workunit.client.0.smithi155.stderr:osd.2 is already in. 2020-03-11T22:44:37.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:318: TEST_divergent_ec: sleep 5 2020-03-11T22:44:42.361 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:44:42.362 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:322: TEST_divergent_ec: objname=existing_4 2020-03-11T22:44:42.362 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:323: TEST_divergent_ec: echo 'writing non-divergent object existing_4' 2020-03-11T22:44:42.362 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:324: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:44:42.362 INFO:tasks.workunit.client.0.smithi155.stdout:writing non-divergent object existing_4 2020-03-11T22:44:42.692 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:44:42.692 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 100 0 100 0 0 48300 0 0 16 16 active+undersized+degraded 2020-03-11T22:44:37.459124+0000 27'100 38:122 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:44:42.692 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:44:42.692 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:44:42.694 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:44:42.702 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:325: TEST_divergent_ec: rados -p test put existing_4 /etc/resolv.conf 2020-03-11T22:44:42.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:44:42.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:327: TEST_divergent_ec: wait_for_clean 2020-03-11T22:44:42.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:44:42.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:44:42.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:44:42.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:44:42.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:44:42.754 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:44:42.755 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:44:42.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:44:42.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:44:42.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:44:42.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:44:42.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:44:42.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:44:42.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:44:42.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:44:43.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:44:43.157 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:44:43.157 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:44:43.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:44:43.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:43.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:44:43.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888068 2020-03-11T22:44:43.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888068' 2020-03-11T22:44:43.286 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:43.286 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:44:43.377 INFO:tasks.workunit.client.0.smithi155.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:44:43.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:44:43.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888068 1-' 2020-03-11T22:44:43.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:44:43.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:44:43.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=150323855364 2020-03-11T22:44:43.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-146028888068 1- 2-150323855364' 2020-03-11T22:44:43.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:43.516 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-146028888068 2020-03-11T22:44:43.516 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:43.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:44:43.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-146028888068 2020-03-11T22:44:43.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:43.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888068 2020-03-11T22:44:43.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 146028888068' 2020-03-11T22:44:43.518 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:43.518 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 146028888068 2020-03-11T22:44:43.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888066 -lt 146028888068 2020-03-11T22:44:43.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:44:44.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:44:44.861 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:44:45.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888069 -lt 146028888068 2020-03-11T22:44:45.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:45.209 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:44:45.209 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:45.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:44:45.210 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:44:45.210 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:45.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:44:45.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:44:45.211 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 2020-03-11T22:44:45.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:44:45.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705683 -lt 2020-03-11T22:44:45.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705683: unary operator expected 2020-03-11T22:44:45.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:44:45.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-150323855364 2020-03-11T22:44:45.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:44:45.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:44:45.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-150323855364 2020-03-11T22:44:45.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:44:45.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=150323855364 2020-03-11T22:44:45.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 150323855364' 2020-03-11T22:44:45.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:44:45.559 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 150323855364 2020-03-11T22:44:45.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 150323855364 -lt 150323855364 2020-03-11T22:44:45.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:44:45.911 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:45.911 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:46.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:44:46.361 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:46.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:46.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:46.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:46.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:46.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:46.362 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:46.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:46.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:46.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:46.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:47.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:47.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:44:47.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:44:47.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:44:47.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:47.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:44:47.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:47.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:47.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:47.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:47.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:47.234 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:47.235 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:47.235 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:47.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:47.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:47.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:47.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:48.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:48.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:48.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:48.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:48.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:48.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:48.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:48.012 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:48.012 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:48.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:48.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:48.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:44:48.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:48.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:44:48.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:48.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:48.725 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:48.725 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:48.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:48.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:48.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:48.726 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:49.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:49.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:49.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:49.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:49.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:49.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:49.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:49.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:49.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:49.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:49.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:49.504 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:49.504 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:49.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:49.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:49.943 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:44:49.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:49.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:44:50.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:50.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:50.345 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:50.345 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:50.345 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:50.345 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:50.345 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:50.346 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:50.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:50.709 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:50.709 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:50.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:51.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:51.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:51.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:51.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:51.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:51.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:51.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:51.190 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:51.190 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:51.634 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:51.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:51.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:44:51.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:51.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:44:52.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:52.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:52.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:52.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:52.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:52.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:52.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:52.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:52.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:52.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:52.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:52.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:53.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:53.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:53.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:53.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:53.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:53.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:53.224 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:53.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:53.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:53.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:53.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:53.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:44:53.669 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:53.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:44:55.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:55.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:55.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:55.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:55.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:55.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:55.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:55.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:44:55.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:44:55.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:44:55.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:44:55.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:44:56.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:44:56.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:44:56.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:44:56.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:44:56.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:44:56.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:44:56.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:44:56.070 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:44:56.070 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:44:56.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:44:56.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:44:56.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:44:56.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:44:56.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:44:59.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:44:59.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:44:59.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:44:59.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:44:59.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:44:59.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:44:59.714 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:44:59.714 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:00.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:00.052 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:00.052 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:00.052 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:00.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:00.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:00.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:00.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:00.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:00.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:00.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:00.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:00.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:00.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:00.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:00.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:45:00.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:00.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:45:07.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:07.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:07.368 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:07.368 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:07.368 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:07.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:07.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:07.369 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:07.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:07.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:07.714 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:07.714 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:08.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:08.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:08.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:08.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:08.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:08.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:08.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:08.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:08.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:08.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:08.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:08.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:45:08.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:08.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:45:15.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:15.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:15.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:15.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:15.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:15.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:15.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:15.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:16.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:16.241 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:16.241 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:16.241 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:16.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:16.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:16.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:16.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:16.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:16.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:16.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:16.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:16.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:17.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:17.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:17.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:45:17.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:45:17.590 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:45:17.590 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_fingerprint": "2b255bec-2463-4ae3-b895-035b99acf8bd", 2020-03-11T22:45:17.590 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:45:17.591 INFO:tasks.workunit.client.0.smithi155.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:45:17.591 INFO:tasks.workunit.client.0.smithi155.stdout: "timestamp": "2020-03-11T22:45:17.587114+0000", 2020-03-11T22:45:17.591 INFO:tasks.workunit.client.0.smithi155.stdout: "tag": "", 2020-03-11T22:45:17.591 INFO:tasks.workunit.client.0.smithi155.stdout: "health": { 2020-03-11T22:45:17.591 INFO:tasks.workunit.client.0.smithi155.stdout: "status": "HEALTH_WARN", 2020-03-11T22:45:17.591 INFO:tasks.workunit.client.0.smithi155.stdout: "checks": { 2020-03-11T22:45:17.591 INFO:tasks.workunit.client.0.smithi155.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:45:17.591 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:17.592 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:45:17.592 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:45:17.592 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 17 2020-03-11T22:45:17.592 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.592 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [], 2020-03-11T22:45:17.592 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:45:17.592 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.593 INFO:tasks.workunit.client.0.smithi155.stdout: "OSD_DOWN": { 2020-03-11T22:45:17.593 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:17.593 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:45:17.593 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 osds down", 2020-03-11T22:45:17.593 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:45:17.593 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.593 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:45:17.594 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.594 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "osd.1 (root=default,host=smithi155) is down" 2020-03-11T22:45:17.594 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.594 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.594 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:45:17.594 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.595 INFO:tasks.workunit.client.0.smithi155.stdout: "PG_DEGRADED": { 2020-03-11T22:45:17.595 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:17.596 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:45:17.596 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:45:17.596 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:45:17.596 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.596 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:45:17.597 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.597 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:45:17.597 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.597 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.597 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:45:17.597 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.597 INFO:tasks.workunit.client.0.smithi155.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:45:17.598 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:17.598 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:45:17.598 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:45:17.598 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:45:17.598 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.598 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:45:17.598 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.599 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:45:17.599 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.599 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.599 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:45:17.599 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.599 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.599 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:45:17.600 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.600 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.600 INFO:tasks.workunit.client.0.smithi155.stdout: "mutes": [] 2020-03-11T22:45:17.600 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.600 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_first_committed": 1, 2020-03-11T22:45:17.600 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_last_committed": 1, 2020-03-11T22:45:17.600 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap": { 2020-03-11T22:45:17.600 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:45:17.600 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:45:17.601 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:43:29.340201Z", 2020-03-11T22:45:17.601 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:43:29.340201Z", 2020-03-11T22:45:17.601 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release": 15, 2020-03-11T22:45:17.601 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:45:17.601 INFO:tasks.workunit.client.0.smithi155.stdout: "features": { 2020-03-11T22:45:17.601 INFO:tasks.workunit.client.0.smithi155.stdout: "persistent": [ 2020-03-11T22:45:17.601 INFO:tasks.workunit.client.0.smithi155.stdout: "kraken", 2020-03-11T22:45:17.601 INFO:tasks.workunit.client.0.smithi155.stdout: "luminous", 2020-03-11T22:45:17.602 INFO:tasks.workunit.client.0.smithi155.stdout: "mimic", 2020-03-11T22:45:17.602 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap-prune", 2020-03-11T22:45:17.602 INFO:tasks.workunit.client.0.smithi155.stdout: "nautilus", 2020-03-11T22:45:17.602 INFO:tasks.workunit.client.0.smithi155.stdout: "octopus" 2020-03-11T22:45:17.602 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.602 INFO:tasks.workunit.client.0.smithi155.stdout: "optional": [] 2020-03-11T22:45:17.602 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.603 INFO:tasks.workunit.client.0.smithi155.stdout: "mons": [ 2020-03-11T22:45:17.603 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.603 INFO:tasks.workunit.client.0.smithi155.stdout: "rank": 0, 2020-03-11T22:45:17.603 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "a", 2020-03-11T22:45:17.603 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:45:17.603 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.603 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.603 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.604 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:45:17.604 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 0 2020-03-11T22:45:17.604 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.604 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.604 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.604 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:45:17.604 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:45:17.604 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 0, 2020-03-11T22:45:17.605 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 0 2020-03-11T22:45:17.605 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.605 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.605 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.605 INFO:tasks.workunit.client.0.smithi155.stdout: "quorum": [ 2020-03-11T22:45:17.605 INFO:tasks.workunit.client.0.smithi155.stdout: 0 2020-03-11T22:45:17.605 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.605 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap": { 2020-03-11T22:45:17.606 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 38, 2020-03-11T22:45:17.606 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:45:17.606 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:43:30.109657+0000", 2020-03-11T22:45:17.606 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:44:37.236572+0000", 2020-03-11T22:45:17.606 INFO:tasks.workunit.client.0.smithi155.stdout: "last_up_change": "2020-03-11T22:44:35.637584+0000", 2020-03-11T22:45:17.606 INFO:tasks.workunit.client.0.smithi155.stdout: "last_in_change": "2020-03-11T22:44:37.061242+0000", 2020-03-11T22:45:17.606 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:45:17.606 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_num": 5800384, 2020-03-11T22:45:17.606 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_set": [ 2020-03-11T22:45:17.607 INFO:tasks.workunit.client.0.smithi155.stdout: "nodown", 2020-03-11T22:45:17.607 INFO:tasks.workunit.client.0.smithi155.stdout: "noin", 2020-03-11T22:45:17.607 INFO:tasks.workunit.client.0.smithi155.stdout: "noout", 2020-03-11T22:45:17.607 INFO:tasks.workunit.client.0.smithi155.stdout: "pglog_hardlimit", 2020-03-11T22:45:17.607 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snapdirs", 2020-03-11T22:45:17.607 INFO:tasks.workunit.client.0.smithi155.stdout: "recovery_deletes", 2020-03-11T22:45:17.607 INFO:tasks.workunit.client.0.smithi155.stdout: "sortbitwise" 2020-03-11T22:45:17.607 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.608 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_version": 8, 2020-03-11T22:45:17.608 INFO:tasks.workunit.client.0.smithi155.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:45:17.608 INFO:tasks.workunit.client.0.smithi155.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:45:17.608 INFO:tasks.workunit.client.0.smithi155.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:45:17.608 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_snapshot": "", 2020-03-11T22:45:17.608 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_max": 1, 2020-03-11T22:45:17.608 INFO:tasks.workunit.client.0.smithi155.stdout: "max_osd": 3, 2020-03-11T22:45:17.608 INFO:tasks.workunit.client.0.smithi155.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:45:17.609 INFO:tasks.workunit.client.0.smithi155.stdout: "min_compat_client": "jewel", 2020-03-11T22:45:17.609 INFO:tasks.workunit.client.0.smithi155.stdout: "require_osd_release": "octopus", 2020-03-11T22:45:17.609 INFO:tasks.workunit.client.0.smithi155.stdout: "pools": [ 2020-03-11T22:45:17.609 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.609 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:45:17.609 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_name": "test", 2020-03-11T22:45:17.609 INFO:tasks.workunit.client.0.smithi155.stdout: "create_time": "2020-03-11T22:43:52.806296+0000", 2020-03-11T22:45:17.609 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": 16389, 2020-03-11T22:45:17.610 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:45:17.610 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 3, 2020-03-11T22:45:17.610 INFO:tasks.workunit.client.0.smithi155.stdout: "size": 3, 2020-03-11T22:45:17.610 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 2, 2020-03-11T22:45:17.610 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_rule": 1, 2020-03-11T22:45:17.610 INFO:tasks.workunit.client.0.smithi155.stdout: "object_hash": 2, 2020-03-11T22:45:17.610 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:45:17.611 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num": 1, 2020-03-11T22:45:17.611 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num": 1, 2020-03-11T22:45:17.611 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num_target": 1, 2020-03-11T22:45:17.611 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_target": 1, 2020-03-11T22:45:17.611 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_pending": 1, 2020-03-11T22:45:17.611 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pg_merge_meta": { 2020-03-11T22:45:17.611 INFO:tasks.workunit.client.0.smithi155.stdout: "source_pgid": "0.0", 2020-03-11T22:45:17.611 INFO:tasks.workunit.client.0.smithi155.stdout: "ready_epoch": 0, 2020-03-11T22:45:17.612 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_started": 0, 2020-03-11T22:45:17.612 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_clean": 0, 2020-03-11T22:45:17.612 INFO:tasks.workunit.client.0.smithi155.stdout: "source_version": "0'0", 2020-03-11T22:45:17.612 INFO:tasks.workunit.client.0.smithi155.stdout: "target_version": "0'0" 2020-03-11T22:45:17.612 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.612 INFO:tasks.workunit.client.0.smithi155.stdout: "last_change": "28", 2020-03-11T22:45:17.612 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend": "0", 2020-03-11T22:45:17.613 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:45:17.613 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:45:17.613 INFO:tasks.workunit.client.0.smithi155.stdout: "auid": 0, 2020-03-11T22:45:17.613 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_mode": "pool", 2020-03-11T22:45:17.613 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_seq": 1, 2020-03-11T22:45:17.613 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_epoch": 28, 2020-03-11T22:45:17.613 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_snaps": [ 2020-03-11T22:45:17.613 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.614 INFO:tasks.workunit.client.0.smithi155.stdout: "snapid": 1, 2020-03-11T22:45:17.614 INFO:tasks.workunit.client.0.smithi155.stdout: "stamp": "2020-03-11T22:44:18.508398+0000", 2020-03-11T22:45:17.614 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "snap1" 2020-03-11T22:45:17.614 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.614 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.614 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps": "[]", 2020-03-11T22:45:17.614 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_bytes": 0, 2020-03-11T22:45:17.615 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_objects": 0, 2020-03-11T22:45:17.615 INFO:tasks.workunit.client.0.smithi155.stdout: "tiers": [], 2020-03-11T22:45:17.615 INFO:tasks.workunit.client.0.smithi155.stdout: "tier_of": -1, 2020-03-11T22:45:17.615 INFO:tasks.workunit.client.0.smithi155.stdout: "read_tier": -1, 2020-03-11T22:45:17.615 INFO:tasks.workunit.client.0.smithi155.stdout: "write_tier": -1, 2020-03-11T22:45:17.615 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_mode": "none", 2020-03-11T22:45:17.616 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_bytes": 0, 2020-03-11T22:45:17.616 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_objects": 0, 2020-03-11T22:45:17.616 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:45:17.616 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:45:17.616 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:45:17.616 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_flush_age": 0, 2020-03-11T22:45:17.616 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_evict_age": 0, 2020-03-11T22:45:17.616 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:45:17.617 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_params": { 2020-03-11T22:45:17.617 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "none" 2020-03-11T22:45:17.617 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.617 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_period": 0, 2020-03-11T22:45:17.617 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_count": 0, 2020-03-11T22:45:17.617 INFO:tasks.workunit.client.0.smithi155.stdout: "use_gmt_hitset": true, 2020-03-11T22:45:17.617 INFO:tasks.workunit.client.0.smithi155.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:45:17.617 INFO:tasks.workunit.client.0.smithi155.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:45:17.618 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:45:17.618 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:45:17.618 INFO:tasks.workunit.client.0.smithi155.stdout: "grade_table": [], 2020-03-11T22:45:17.619 INFO:tasks.workunit.client.0.smithi155.stdout: "stripe_width": 8192, 2020-03-11T22:45:17.619 INFO:tasks.workunit.client.0.smithi155.stdout: "expected_num_objects": 0, 2020-03-11T22:45:17.619 INFO:tasks.workunit.client.0.smithi155.stdout: "fast_read": false, 2020-03-11T22:45:17.619 INFO:tasks.workunit.client.0.smithi155.stdout: "options": {}, 2020-03-11T22:45:17.620 INFO:tasks.workunit.client.0.smithi155.stdout: "application_metadata": {} 2020-03-11T22:45:17.620 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.620 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.620 INFO:tasks.workunit.client.0.smithi155.stdout: "osds": [ 2020-03-11T22:45:17.620 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.620 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:45:17.621 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "e0e5724e-7440-465c-9db6-82e8c737bcc8", 2020-03-11T22:45:17.621 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:45:17.621 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:45:17.621 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:45:17.621 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:45:17.621 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 5, 2020-03-11T22:45:17.621 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 27, 2020-03-11T22:45:17.621 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 34, 2020-03-11T22:45:17.621 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 37, 2020-03-11T22:45:17.622 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 29, 2020-03-11T22:45:17.622 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:45:17.622 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:45:17.622 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.623 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.623 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.624 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:45:17.624 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48706 2020-03-11T22:45:17.624 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.624 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.624 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:17.624 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:45:17.624 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48706 2020-03-11T22:45:17.625 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.625 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.625 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.625 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:45:17.625 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.625 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.625 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.626 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:45:17.626 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48706 2020-03-11T22:45:17.626 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.626 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.626 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:17.626 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:45:17.627 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48706 2020-03-11T22:45:17.627 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.627 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.627 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.627 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:45:17.627 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.627 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.627 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.628 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:45:17.628 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48706 2020-03-11T22:45:17.628 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.628 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.628 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:17.628 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:45:17.628 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48706 2020-03-11T22:45:17.628 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.629 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.629 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.629 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:45:17.629 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.629 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.629 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.629 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:45:17.629 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48706 2020-03-11T22:45:17.630 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.630 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.630 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:17.630 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:45:17.630 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48706 2020-03-11T22:45:17.630 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.630 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.630 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.630 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6803/48706", 2020-03-11T22:45:17.631 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6805/48706", 2020-03-11T22:45:17.631 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6809/48706", 2020-03-11T22:45:17.631 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6807/48706", 2020-03-11T22:45:17.631 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:45:17.631 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:45:17.631 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:45:17.632 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.632 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.632 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.632 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:45:17.632 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "7b6947d8-603c-41a4-abcb-8b50325a2007", 2020-03-11T22:45:17.632 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 0, 2020-03-11T22:45:17.632 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:45:17.632 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:45:17.633 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:45:17.633 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 0, 2020-03-11T22:45:17.633 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 0, 2020-03-11T22:45:17.633 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 9, 2020-03-11T22:45:17.633 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 23, 2020-03-11T22:45:17.633 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 30, 2020-03-11T22:45:17.633 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:45:17.633 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:45:17.633 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.634 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.634 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.634 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:45:17.634 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:17.634 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.634 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.634 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:17.634 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:45:17.635 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:17.635 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.635 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.635 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.635 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:45:17.635 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.635 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.636 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.636 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:45:17.636 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:17.636 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.636 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.636 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:17.636 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:45:17.637 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:17.637 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.637 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.637 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.637 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:45:17.637 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.637 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.638 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.638 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:45:17.638 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:17.638 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.638 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.638 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:17.639 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:45:17.639 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:17.639 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.639 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.639 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.639 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:45:17.639 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.640 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.640 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.640 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:45:17.640 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:17.640 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.640 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.640 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:17.641 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:45:17.641 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:17.641 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.641 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.641 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.641 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/43970", 2020-03-11T22:45:17.641 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/43970", 2020-03-11T22:45:17.642 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43970", 2020-03-11T22:45:17.642 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43970", 2020-03-11T22:45:17.642 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:45:17.642 INFO:tasks.workunit.client.0.smithi155.stdout: "exists" 2020-03-11T22:45:17.642 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.642 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.642 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.643 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:45:17.643 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "797c47f4-417e-4663-8ca9-e4cc78330dd9", 2020-03-11T22:45:17.643 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:45:17.643 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:45:17.643 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:45:17.643 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:45:17.643 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 14, 2020-03-11T22:45:17.644 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 27, 2020-03-11T22:45:17.644 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 35, 2020-03-11T22:45:17.644 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 0, 2020-03-11T22:45:17.644 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 31, 2020-03-11T22:45:17.644 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:45:17.644 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:45:17.644 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.644 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.645 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.645 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:45:17.645 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48900 2020-03-11T22:45:17.645 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.645 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.645 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:17.645 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:45:17.645 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48900 2020-03-11T22:45:17.646 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.646 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.646 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.646 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:45:17.646 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.646 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:45:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48900 2020-03-11T22:45:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:45:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48900 2020-03-11T22:45:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:45:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:45:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48900 2020-03-11T22:45:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:45:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48900 2020-03-11T22:45:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:45:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:17.651 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:45:17.651 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48900 2020-03-11T22:45:17.651 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.651 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.651 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:17.651 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:45:17.651 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 48900 2020-03-11T22:45:17.652 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.652 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.652 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.652 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/48900", 2020-03-11T22:45:17.652 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/48900", 2020-03-11T22:45:17.652 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/48900", 2020-03-11T22:45:17.652 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/48900", 2020-03-11T22:45:17.652 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:45:17.653 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:45:17.653 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:45:17.653 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.653 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.653 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.653 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_xinfo": [ 2020-03-11T22:45:17.653 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.654 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:45:17.654 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:44:29.396677+0000", 2020-03-11T22:45:17.654 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:45:17.654 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:45:17.654 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:45:17.654 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:45:17.654 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:37.264836+0000", 2020-03-11T22:45:17.655 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:45:17.655 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.655 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.655 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:45:17.655 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:44:29.934725+0000", 2020-03-11T22:45:17.655 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:45:17.656 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:45:17.656 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:45:17.656 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:45:17.656 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:43.587931+0000", 2020-03-11T22:45:17.656 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:45:17.656 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.656 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.657 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:45:17.657 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:44:30.470682+0000", 2020-03-11T22:45:17.657 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:45:17.657 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:45:17.657 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:45:17.657 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:45:17.657 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:49.866136+0000", 2020-03-11T22:45:17.658 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:45:17.658 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.658 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.658 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap": [], 2020-03-11T22:45:17.658 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap_items": [], 2020-03-11T22:45:17.658 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_temp": [], 2020-03-11T22:45:17.658 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_temp": [], 2020-03-11T22:45:17.658 INFO:tasks.workunit.client.0.smithi155.stdout: "blacklist": {}, 2020-03-11T22:45:17.659 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profiles": { 2020-03-11T22:45:17.659 INFO:tasks.workunit.client.0.smithi155.stdout: "default": { 2020-03-11T22:45:17.659 INFO:tasks.workunit.client.0.smithi155.stdout: "k": "2", 2020-03-11T22:45:17.659 INFO:tasks.workunit.client.0.smithi155.stdout: "m": "2", 2020-03-11T22:45:17.659 INFO:tasks.workunit.client.0.smithi155.stdout: "plugin": "jerasure", 2020-03-11T22:45:17.659 INFO:tasks.workunit.client.0.smithi155.stdout: "technique": "reed_sol_van" 2020-03-11T22:45:17.659 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.660 INFO:tasks.workunit.client.0.smithi155.stdout: "myprofile": { 2020-03-11T22:45:17.660 INFO:tasks.workunit.client.0.smithi155.stdout: "crush-device-class": "", 2020-03-11T22:45:17.660 INFO:tasks.workunit.client.0.smithi155.stdout: "crush-failure-domain": "osd", 2020-03-11T22:45:17.660 INFO:tasks.workunit.client.0.smithi155.stdout: "crush-root": "default", 2020-03-11T22:45:17.660 INFO:tasks.workunit.client.0.smithi155.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:45:17.660 INFO:tasks.workunit.client.0.smithi155.stdout: "k": "2", 2020-03-11T22:45:17.660 INFO:tasks.workunit.client.0.smithi155.stdout: "m": "1", 2020-03-11T22:45:17.660 INFO:tasks.workunit.client.0.smithi155.stdout: "plugin": "jerasure", 2020-03-11T22:45:17.661 INFO:tasks.workunit.client.0.smithi155.stdout: "technique": "reed_sol_van", 2020-03-11T22:45:17.661 INFO:tasks.workunit.client.0.smithi155.stdout: "w": "8" 2020-03-11T22:45:17.661 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.661 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.661 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps_queue": [], 2020-03-11T22:45:17.661 INFO:tasks.workunit.client.0.smithi155.stdout: "new_removed_snaps": [], 2020-03-11T22:45:17.662 INFO:tasks.workunit.client.0.smithi155.stdout: "new_purged_snaps": [], 2020-03-11T22:45:17.662 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_node_flags": {}, 2020-03-11T22:45:17.662 INFO:tasks.workunit.client.0.smithi155.stdout: "device_class_flags": {} 2020-03-11T22:45:17.663 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.663 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_metadata": [ 2020-03-11T22:45:17.663 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.663 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:45:17.663 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:45:17.663 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6804/48706,v1:127.0.0.1:6805/48706]", 2020-03-11T22:45:17.663 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:45:17.663 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:45:17.664 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:45:17.664 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:45:17.664 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:45:17.664 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:45:17.664 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:45:17.664 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:45:17.664 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:45:17.665 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:45:17.665 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:45:17.665 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:45:17.665 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:45:17.665 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:45:17.665 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:45:17.665 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:45:17.665 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:45:17.666 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:45:17.666 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:45:17.666 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:45:17.666 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:45:17.666 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:45:17.666 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:45:17.666 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:45:17.667 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:45:17.667 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6802/48706,v1:127.0.0.1:6803/48706]", 2020-03-11T22:45:17.667 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:45:17.667 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/48706,v1:127.0.0.1:6809/48706]", 2020-03-11T22:45:17.667 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/48706,v1:127.0.0.1:6807/48706]", 2020-03-11T22:45:17.667 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:45:17.667 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:45:17.668 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:45:17.668 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:45:17.668 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:45:17.668 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:45:17.668 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:45:17.668 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:45:17.668 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:45:17.669 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:45:17.669 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:45:17.669 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:45:17.669 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.669 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.669 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:45:17.670 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:45:17.670 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/43970,v1:127.0.0.1:6813/43970]", 2020-03-11T22:45:17.671 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:45:17.671 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:45:17.671 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:45:17.671 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:45:17.671 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:45:17.671 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:45:17.672 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:45:17.672 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:45:17.672 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:45:17.672 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:45:17.672 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:45:17.672 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:45:17.672 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:45:17.673 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:45:17.673 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:45:17.673 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:45:17.673 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:45:17.673 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:45:17.673 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:45:17.673 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:45:17.674 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:45:17.674 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:45:17.674 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:45:17.674 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:45:17.674 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:45:17.674 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/43970,v1:127.0.0.1:6811/43970]", 2020-03-11T22:45:17.674 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:45:17.675 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43970,v1:127.0.0.1:6817/43970]", 2020-03-11T22:45:17.675 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43970,v1:127.0.0.1:6815/43970]", 2020-03-11T22:45:17.675 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:45:17.675 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:45:17.675 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:45:17.675 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:45:17.675 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:45:17.676 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:45:17.676 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:45:17.676 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:45:17.676 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:45:17.676 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:45:17.676 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:45:17.676 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:45:17.677 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.677 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.677 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:45:17.677 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:45:17.677 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/48900,v1:127.0.0.1:6813/48900]", 2020-03-11T22:45:17.677 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:45:17.677 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:45:17.677 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:45:17.678 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:45:17.678 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:45:17.678 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:45:17.678 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:45:17.678 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:45:17.678 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:45:17.678 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:45:17.679 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:45:17.679 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:45:17.679 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:45:17.679 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:45:17.679 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:45:17.679 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:45:17.679 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:45:17.680 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:45:17.680 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:45:17.680 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:45:17.680 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:45:17.680 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:45:17.680 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:45:17.680 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:45:17.681 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:45:17.681 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/48900,v1:127.0.0.1:6811/48900]", 2020-03-11T22:45:17.681 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:45:17.681 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/48900,v1:127.0.0.1:6817/48900]", 2020-03-11T22:45:17.681 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/48900,v1:127.0.0.1:6815/48900]", 2020-03-11T22:45:17.681 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:45:17.681 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:45:17.682 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:45:17.682 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:45:17.682 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:45:17.682 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:45:17.682 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:45:17.682 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:45:17.682 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:45:17.683 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:45:17.683 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:45:17.683 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:45:17.683 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.683 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.683 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_first_committed": 1, 2020-03-11T22:45:17.683 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_last_committed": 38, 2020-03-11T22:45:17.683 INFO:tasks.workunit.client.0.smithi155.stdout: "crushmap": { 2020-03-11T22:45:17.684 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": [ 2020-03-11T22:45:17.684 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.684 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:45:17.684 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.0", 2020-03-11T22:45:17.684 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:45:17.684 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.684 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.684 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:45:17.685 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.1", 2020-03-11T22:45:17.685 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:45:17.685 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.685 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.685 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:45:17.685 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.2", 2020-03-11T22:45:17.685 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:45:17.686 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.686 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.686 INFO:tasks.workunit.client.0.smithi155.stdout: "types": [ 2020-03-11T22:45:17.686 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.686 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 0, 2020-03-11T22:45:17.686 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd" 2020-03-11T22:45:17.686 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.686 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.687 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:45:17.687 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "host" 2020-03-11T22:45:17.687 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.687 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.687 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 2, 2020-03-11T22:45:17.687 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "chassis" 2020-03-11T22:45:17.687 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.688 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.688 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 3, 2020-03-11T22:45:17.688 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "rack" 2020-03-11T22:45:17.688 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.688 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.688 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 4, 2020-03-11T22:45:17.688 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "row" 2020-03-11T22:45:17.689 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.689 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.689 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 5, 2020-03-11T22:45:17.689 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pdu" 2020-03-11T22:45:17.689 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.689 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.689 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 6, 2020-03-11T22:45:17.689 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pod" 2020-03-11T22:45:17.690 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.690 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.690 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 7, 2020-03-11T22:45:17.690 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "room" 2020-03-11T22:45:17.690 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.690 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.690 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 8, 2020-03-11T22:45:17.691 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "datacenter" 2020-03-11T22:45:17.691 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.691 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.691 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 9, 2020-03-11T22:45:17.691 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "zone" 2020-03-11T22:45:17.691 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.691 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.691 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 10, 2020-03-11T22:45:17.692 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "region" 2020-03-11T22:45:17.692 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.692 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.692 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:45:17.692 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "root" 2020-03-11T22:45:17.692 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.692 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.693 INFO:tasks.workunit.client.0.smithi155.stdout: "buckets": [ 2020-03-11T22:45:17.693 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.693 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -1, 2020-03-11T22:45:17.693 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default", 2020-03-11T22:45:17.693 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:45:17.693 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:45:17.693 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:45:17.694 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:45:17.694 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:45:17.694 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:45:17.694 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.694 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:45:17.694 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:45:17.694 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:45:17.695 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.695 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.695 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.695 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.695 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -2, 2020-03-11T22:45:17.695 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default~hdd", 2020-03-11T22:45:17.695 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:45:17.695 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:45:17.696 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:45:17.696 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:45:17.696 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:45:17.696 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:45:17.696 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.696 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:45:17.696 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:45:17.697 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:45:17.697 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.697 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.697 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.697 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.697 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:45:17.697 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155", 2020-03-11T22:45:17.698 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:45:17.698 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:45:17.698 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:45:17.698 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:45:17.698 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:45:17.698 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:45:17.698 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.699 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:45:17.699 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:45:17.699 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:45:17.699 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.699 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.699 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:45:17.699 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:45:17.699 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:45:17.700 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.700 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.700 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:45:17.700 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:45:17.700 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:45:17.700 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.700 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.700 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.701 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.701 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:45:17.701 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155~hdd", 2020-03-11T22:45:17.701 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:45:17.701 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:45:17.701 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:45:17.701 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:45:17.702 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:45:17.703 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi155.stdout: "rules": [ 2020-03-11T22:45:17.704 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_id": 0, 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_name": "replicated_rule", 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi155.stdout: "ruleset": 0, 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 1, 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi155.stdout: "max_size": 10, 2020-03-11T22:45:17.705 INFO:tasks.workunit.client.0.smithi155.stdout: "steps": [ 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "take", 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi155.stdout: "item": -1, 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi155.stdout: "item_name": "default" 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "choose_firstn", 2020-03-11T22:45:17.706 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 0, 2020-03-11T22:45:17.707 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "osd" 2020-03-11T22:45:17.707 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.707 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.707 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "emit" 2020-03-11T22:45:17.707 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.707 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.707 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.708 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.708 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_id": 1, 2020-03-11T22:45:17.708 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_name": "test", 2020-03-11T22:45:17.708 INFO:tasks.workunit.client.0.smithi155.stdout: "ruleset": 1, 2020-03-11T22:45:17.708 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 3, 2020-03-11T22:45:17.708 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 3, 2020-03-11T22:45:17.708 INFO:tasks.workunit.client.0.smithi155.stdout: "max_size": 3, 2020-03-11T22:45:17.708 INFO:tasks.workunit.client.0.smithi155.stdout: "steps": [ 2020-03-11T22:45:17.709 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.709 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:45:17.709 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 5 2020-03-11T22:45:17.709 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.709 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.709 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "set_choose_tries", 2020-03-11T22:45:17.709 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 100 2020-03-11T22:45:17.710 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.710 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.710 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "take", 2020-03-11T22:45:17.710 INFO:tasks.workunit.client.0.smithi155.stdout: "item": -1, 2020-03-11T22:45:17.710 INFO:tasks.workunit.client.0.smithi155.stdout: "item_name": "default" 2020-03-11T22:45:17.710 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.710 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.711 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "choose_indep", 2020-03-11T22:45:17.713 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 0, 2020-03-11T22:45:17.713 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "osd" 2020-03-11T22:45:17.713 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.714 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.714 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "emit" 2020-03-11T22:45:17.714 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.714 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:17.714 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.714 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.714 INFO:tasks.workunit.client.0.smithi155.stdout: "tunables": { 2020-03-11T22:45:17.715 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_tries": 0, 2020-03-11T22:45:17.715 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:45:17.715 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_total_tries": 50, 2020-03-11T22:45:17.715 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:45:17.715 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:45:17.715 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_stable": 1, 2020-03-11T22:45:17.715 INFO:tasks.workunit.client.0.smithi155.stdout: "straw_calc_version": 1, 2020-03-11T22:45:17.716 INFO:tasks.workunit.client.0.smithi155.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:45:17.716 INFO:tasks.workunit.client.0.smithi155.stdout: "profile": "jewel", 2020-03-11T22:45:17.716 INFO:tasks.workunit.client.0.smithi155.stdout: "optimal_tunables": 1, 2020-03-11T22:45:17.716 INFO:tasks.workunit.client.0.smithi155.stdout: "legacy_tunables": 0, 2020-03-11T22:45:17.716 INFO:tasks.workunit.client.0.smithi155.stdout: "minimum_required_version": "jewel", 2020-03-11T22:45:17.716 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables": 1, 2020-03-11T22:45:17.716 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables2": 1, 2020-03-11T22:45:17.716 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v2_rules": 1, 2020-03-11T22:45:17.717 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables3": 1, 2020-03-11T22:45:17.717 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v3_rules": 0, 2020-03-11T22:45:17.717 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v4_buckets": 1, 2020-03-11T22:45:17.717 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables5": 1, 2020-03-11T22:45:17.717 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v5_rules": 0 2020-03-11T22:45:17.717 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.717 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_args": {} 2020-03-11T22:45:17.718 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.718 INFO:tasks.workunit.client.0.smithi155.stdout: "fsmap": { 2020-03-11T22:45:17.718 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:45:17.718 INFO:tasks.workunit.client.0.smithi155.stdout: "default_fscid": -1, 2020-03-11T22:45:17.718 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": { 2020-03-11T22:45:17.718 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": {}, 2020-03-11T22:45:17.718 INFO:tasks.workunit.client.0.smithi155.stdout: "ro_compat": {}, 2020-03-11T22:45:17.719 INFO:tasks.workunit.client.0.smithi155.stdout: "incompat": { 2020-03-11T22:45:17.719 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_1": "base v0.20", 2020-03-11T22:45:17.719 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:45:17.719 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:45:17.719 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:45:17.719 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:45:17.719 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:45:17.720 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_8": "no anchor table", 2020-03-11T22:45:17.720 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_9": "file layout v2", 2020-03-11T22:45:17.720 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:45:17.720 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.720 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.720 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_flags": { 2020-03-11T22:45:17.720 INFO:tasks.workunit.client.0.smithi155.stdout: "enable_multiple": false, 2020-03-11T22:45:17.720 INFO:tasks.workunit.client.0.smithi155.stdout: "ever_enabled_multiple": false 2020-03-11T22:45:17.721 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.721 INFO:tasks.workunit.client.0.smithi155.stdout: "standbys": [], 2020-03-11T22:45:17.721 INFO:tasks.workunit.client.0.smithi155.stdout: "filesystems": [] 2020-03-11T22:45:17.721 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.721 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:45:17.721 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:45:17.721 INFO:tasks.workunit.client.0.smithi155.stdout: "auth": { 2020-03-11T22:45:17.722 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:45:17.722 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 5, 2020-03-11T22:45:17.722 INFO:tasks.workunit.client.0.smithi155.stdout: "num_secrets": 3 2020-03-11T22:45:17.722 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.722 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 1, 2020-03-11T22:45:17.722 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_active": 1, 2020-03-11T22:45:17.722 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_unknown": 0, 2020-03-11T22:45:17.723 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osd": 3, 2020-03-11T22:45:17.723 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_sum": { 2020-03-11T22:45:17.723 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:45:17.723 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 48430, 2020-03-11T22:45:17.723 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 101, 2020-03-11T22:45:17.723 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 1, 2020-03-11T22:45:17.723 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 303, 2020-03-11T22:45:17.724 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:45:17.724 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:45:17.724 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 101, 2020-03-11T22:45:17.724 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:45:17.724 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:45:17.724 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 101, 2020-03-11T22:45:17.724 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:45:17.724 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:45:17.725 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:45:17.725 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 101, 2020-03-11T22:45:17.725 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 101, 2020-03-11T22:45:17.725 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:45:17.725 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:45:17.725 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:45:17.726 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:45:17.726 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:45:17.726 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:45:17.726 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:45:17.726 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:45:17.726 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:45:17.726 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:45:17.726 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:45:17.727 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:45:17.727 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:45:17.727 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:45:17.727 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:45:17.727 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:45:17.727 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:45:17.728 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:45:17.728 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:45:17.728 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:45:17.728 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:45:17.728 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:45:17.728 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:45:17.728 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:45:17.729 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:45:17.729 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.729 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:45:17.729 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:45:17.729 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:45:17.729 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:45:17.730 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 0, 2020-03-11T22:45:17.730 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 0, 2020-03-11T22:45:17.730 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:45:17.730 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:17.730 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:45:17.730 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:45:17.730 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:45:17.731 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.731 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 18, 2020-03-11T22:45:17.731 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 18, 2020-03-11T22:45:17.731 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 3, 2020-03-11T22:45:17.731 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 3, 2020-03-11T22:45:17.731 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 0 2020-03-11T22:45:17.731 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.732 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum": { 2020-03-11T22:45:17.732 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:45:17.732 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:45:17.732 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 3, 2020-03-11T22:45:17.732 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:45:17.732 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:45:17.732 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:45:17.733 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:45:17.733 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3179072, 2020-03-11T22:45:17.733 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 33088, 2020-03-11T22:45:17.733 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 10, 2020-03-11T22:45:17.733 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145717, 2020-03-11T22:45:17.733 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311393728, 2020-03-11T22:45:17.733 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:45:17.734 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:45:17.734 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318867177472, 2020-03-11T22:45:17.734 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:45:17.734 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 33882112, 2020-03-11T22:45:17.734 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 1512544, 2020-03-11T22:45:17.734 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:45:17.734 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:17.735 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:45:17.735 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 10752, 2020-03-11T22:45:17.735 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221214720 2020-03-11T22:45:17.735 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.735 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:45:17.735 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:45:17.735 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:45:17.736 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2020-03-11T22:45:17.736 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:45:17.736 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:45:17.736 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:45:17.736 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.736 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:45:17.736 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:45:17.736 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:45:17.737 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:45:17.737 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:45:17.737 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.737 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:45:17.737 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:45:17.737 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.737 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum_by_class": { 2020-03-11T22:45:17.738 INFO:tasks.workunit.client.0.smithi155.stdout: "hdd": { 2020-03-11T22:45:17.738 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:45:17.738 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:45:17.738 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 3, 2020-03-11T22:45:17.738 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:45:17.738 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:45:17.738 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:45:17.739 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:45:17.739 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3179072, 2020-03-11T22:45:17.739 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 33088, 2020-03-11T22:45:17.739 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 10, 2020-03-11T22:45:17.739 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145717, 2020-03-11T22:45:17.739 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311393728, 2020-03-11T22:45:17.739 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:45:17.740 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:45:17.740 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318867177472, 2020-03-11T22:45:17.740 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:45:17.740 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 33882112, 2020-03-11T22:45:17.740 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 1512544, 2020-03-11T22:45:17.740 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:45:17.740 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:17.740 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:45:17.741 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 10752, 2020-03-11T22:45:17.741 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221214720 2020-03-11T22:45:17.741 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.741 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:45:17.741 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:45:17.741 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:45:17.741 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2020-03-11T22:45:17.742 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:45:17.742 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:45:17.742 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:45:17.742 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.742 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:45:17.742 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:45:17.742 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:45:17.743 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:45:17.743 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:45:17.743 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.743 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:45:17.743 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:45:17.743 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.743 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.743 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_stats": [ 2020-03-11T22:45:17.744 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.744 INFO:tasks.workunit.client.0.smithi155.stdout: "poolid": 1, 2020-03-11T22:45:17.744 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 1, 2020-03-11T22:45:17.744 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:45:17.744 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 48430, 2020-03-11T22:45:17.744 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 101, 2020-03-11T22:45:17.744 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 1, 2020-03-11T22:45:17.745 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 303, 2020-03-11T22:45:17.745 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:45:17.745 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:45:17.745 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 101, 2020-03-11T22:45:17.745 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:45:17.745 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:45:17.745 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 101, 2020-03-11T22:45:17.746 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:45:17.746 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:45:17.746 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:45:17.746 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 101, 2020-03-11T22:45:17.746 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 101, 2020-03-11T22:45:17.746 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:45:17.746 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:45:17.747 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:45:17.747 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:45:17.747 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:45:17.747 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:45:17.747 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:45:17.747 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:45:17.747 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:45:17.748 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:45:17.748 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:45:17.748 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:45:17.748 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:45:17.748 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:45:17.748 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:45:17.748 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:45:17.749 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:45:17.749 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:45:17.749 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:45:17.749 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:45:17.749 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:45:17.749 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:45:17.749 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:45:17.749 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:45:17.750 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:45:17.750 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.750 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:45:17.750 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:45:17.750 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:45:17.750 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:45:17.750 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 19857408, 2020-03-11T22:45:17.751 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 1241088, 2020-03-11T22:45:17.751 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:45:17.751 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:17.751 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:45:17.751 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:45:17.751 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:45:17.751 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.752 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 18, 2020-03-11T22:45:17.752 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 18, 2020-03-11T22:45:17.752 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 3, 2020-03-11T22:45:17.752 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 3, 2020-03-11T22:45:17.752 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 3 2020-03-11T22:45:17.752 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.752 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.752 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_stats": [ 2020-03-11T22:45:17.753 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.753 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:45:17.753 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 146028888075 2020-03-11T22:45:17.753 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.753 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.753 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:45:17.753 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 38654705683 2020-03-11T22:45:17.754 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.754 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.754 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:45:17.754 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 150323855371 2020-03-11T22:45:17.754 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.754 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.754 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_state": [ 2020-03-11T22:45:17.755 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.755 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "active+undersized+degraded", 2020-03-11T22:45:17.755 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 1 2020-03-11T22:45:17.755 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.755 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.755 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_osd": [ 2020-03-11T22:45:17.755 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.756 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:45:17.756 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:45:17.756 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 1, 2020-03-11T22:45:17.756 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:17.756 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.756 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.756 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2147483647, 2020-03-11T22:45:17.756 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:45:17.757 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 1, 2020-03-11T22:45:17.757 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:17.757 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.757 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.757 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:45:17.757 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:45:17.757 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 0, 2020-03-11T22:45:17.758 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:17.758 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.758 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.758 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:45:17.758 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 1, 2020-03-11T22:45:17.758 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 1, 2020-03-11T22:45:17.758 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:17.759 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.759 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.759 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": [ 2020-03-11T22:45:17.759 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:17.759 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:45:17.759 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": {} 2020-03-11T22:45:17.759 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.759 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:17.760 INFO:tasks.workunit.client.0.smithi155.stdout: "servicemap": { 2020-03-11T22:45:17.760 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:45:17.760 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:43:30.109994+0000", 2020-03-11T22:45:17.760 INFO:tasks.workunit.client.0.smithi155.stdout: "services": {} 2020-03-11T22:45:17.760 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:17.760 INFO:tasks.workunit.client.0.smithi155.stdout: "paxos": { 2020-03-11T22:45:17.760 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:45:17.760 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 326, 2020-03-11T22:45:17.761 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pn": 0, 2020-03-11T22:45:17.761 INFO:tasks.workunit.client.0.smithi155.stdout: "accepted_pn": 0 2020-03-11T22:45:17.761 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:17.761 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:45:17.763 INFO:tasks.workunit.client.0.smithi155.stderr:report 1148873079 2020-03-11T22:45:17.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:45:17.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:45:17.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.0 2020-03-11T22:45:17.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:17.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:17.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:17.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:17.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:17.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:17.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 0 --op log --pgid 1.0 2020-03-11T22:45:17.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:45:17.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:45:17.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=0 2020-03-11T22:45:17.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:45:17.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/0 2020-03-11T22:45:17.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/0 --op log --pgid 1.0 2020-03-11T22:45:18.192 INFO:tasks.workunit.client.0.smithi155.stderr:PG '1.0' not found 2020-03-11T22:45:18.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:45:18.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 0 2020-03-11T22:45:18.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:45:18.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:18.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:45:18.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:18.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/0 2020-03-11T22:45:18.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:45:18.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:18.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:18.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:18.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/0' 2020-03-11T22:45:18.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/0/journal' 2020-03-11T22:45:18.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:18.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:18.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:45:18.712 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:18.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:18.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:18.713 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:18.713 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:18.713 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:45:18.713 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:45:18.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:45:18.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:18.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:45:18.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:45:18.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:18.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:18.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:18.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:18.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:18.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/0 2020-03-11T22:45:18.716 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:45:18.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:45:18.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/0 --osd-journal=td/divergent-priors/0/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:18.716 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/0/whoami 2020-03-11T22:45:18.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:45:18.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:18.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:18.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:18.740 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:18.738+0000 7f3152659ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:18.743 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:18.740+0000 7f3152659ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:18.745 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:18.743+0000 7f3152659ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:19.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:45:19.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:19.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:45:19.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:19.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:19.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:19.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:19.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:19.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:45:19.070 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:45:19.261 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:19.258+0000 7f3152659ec0 -1 Falling back to public interface 2020-03-11T22:45:19.413 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 34 up_thru 37 down_at 29 last_clean_interval [5,27) [v2:127.0.0.1:6802/48706,v1:127.0.0.1:6803/48706] [v2:127.0.0.1:6804/48706,v1:127.0.0.1:6805/48706] exists,up e0e5724e-7440-465c-9db6-82e8c737bcc8 2020-03-11T22:45:19.413 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:19.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:19.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:19.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:330: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:45:19.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:332: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.2 2020-03-11T22:45:19.414 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:45:19.414 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:45:19.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:45:19.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:45:19.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:45:19.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:45:19.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:333: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 2 --op log --pgid 1.0 2020-03-11T22:45:19.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:45:19.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:45:19.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=2 2020-03-11T22:45:19.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:45:19.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/2 2020-03-11T22:45:19.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/2 --op log --pgid 1.0 2020-03-11T22:45:20.271 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:20.268+0000 7f3152659ec0 -1 osd.0 38 log_to_monitors {default=true} 2020-03-11T22:45:20.336 INFO:tasks.workunit.client.0.smithi155.stderr:PG '1.0' not found 2020-03-11T22:45:20.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:45:20.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:334: TEST_divergent_ec: activate_osd td/divergent-priors 2 2020-03-11T22:45:20.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:45:20.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:20.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:45:20.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:20.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/2 2020-03-11T22:45:20.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:45:20.856 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:20.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:20.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:20.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/2' 2020-03-11T22:45:20.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/2/journal' 2020-03-11T22:45:20.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:20.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:20.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:45:20.858 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:20.858 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:20.858 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:20.858 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:20.858 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:20.858 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:45:20.858 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:45:20.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:45:20.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:20.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:45:20.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:45:20.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:20.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:20.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:20.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:20.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:20.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/2 2020-03-11T22:45:20.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:45:20.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/2 --osd-journal=td/divergent-priors/2/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:20.860 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:45:20.861 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/2/whoami 2020-03-11T22:45:20.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:45:20.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:20.861 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:20.862 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:20.884 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:20.881+0000 7fb8f6347ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:20.886 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:20.883+0000 7fb8f6347ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:20.888 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:20.885+0000 7fb8f6347ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:21.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:45:21.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:45:21.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:45:21.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:45:21.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:45:21.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:45:21.215 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:45:21.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:45:21.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:45:21.216 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:45:21.415 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:21.400+0000 7fb8f6347ec0 -1 Falling back to public interface 2020-03-11T22:45:21.570 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 35 up_thru 0 down_at 31 last_clean_interval [14,27) [v2:127.0.0.1:6810/48900,v1:127.0.0.1:6811/48900] [v2:127.0.0.1:6812/48900,v1:127.0.0.1:6813/48900] exists,up 797c47f4-417e-4663-8ca9-e4cc78330dd9 2020-03-11T22:45:21.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:45:21.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:45:21.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:45:21.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:336: TEST_divergent_ec: _objectstore_tool_nodown td/divergent-priors 1 --op log --pgid 1.0 2020-03-11T22:45:21.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/divergent-priors 2020-03-11T22:45:21.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:45:21.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=1 2020-03-11T22:45:21.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:45:21.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/divergent-priors/1 2020-03-11T22:45:21.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/divergent-priors/1 --op log --pgid 1.0 2020-03-11T22:45:22.242 INFO:tasks.workunit.client.0.smithi155.stderr:PG '1.0' not found 2020-03-11T22:45:22.313 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:22.311+0000 7fb8f6347ec0 -1 osd.2 38 log_to_monitors {default=true} 2020-03-11T22:45:22.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: _objectstore_tool_nodown: return 1 2020-03-11T22:45:22.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: WAIT_FOR_CLEAN_TIMEOUT=20 2020-03-11T22:45:22.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:338: TEST_divergent_ec: wait_for_clean 2020-03-11T22:45:22.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:45:22.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:45:22.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:45:22.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:45:22.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 20 .1 2020-03-11T22:45:22.761 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:45:22.761 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:45:22.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:45:22.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:45:22.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:45:22.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:45:22.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:45:22.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:45:22.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:45:22.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:45:23.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:45:23.173 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:45:23.174 INFO:tasks.workunit.client.0.smithi155.stderr:2' 2020-03-11T22:45:23.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:45:23.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:23.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:45:23.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=171798691843 2020-03-11T22:45:23.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-171798691843' 2020-03-11T22:45:23.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:23.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:45:23.393 INFO:tasks.workunit.client.0.smithi155.stderr:Error ENXIO: problem getting command descriptions from osd.1 2020-03-11T22:45:23.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq= 2020-03-11T22:45:23.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-171798691843 1-' 2020-03-11T22:45:23.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:45:23.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:45:23.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=184683593731 2020-03-11T22:45:23.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-171798691843 1- 2-184683593731' 2020-03-11T22:45:23.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:23.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-171798691843 2020-03-11T22:45:23.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:23.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:45:23.532 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-171798691843 2020-03-11T22:45:23.532 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:23.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=171798691843 2020-03-11T22:45:23.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 171798691843' 2020-03-11T22:45:23.533 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:23.533 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 171798691843 2020-03-11T22:45:23.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 171798691843 2020-03-11T22:45:23.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:45:24.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:45:24.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:45:25.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 171798691843 -lt 171798691843 2020-03-11T22:45:25.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:25.245 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1- 2020-03-11T22:45:25.245 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:25.246 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:45:25.246 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1- 2020-03-11T22:45:25.247 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:25.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq= 2020-03-11T22:45:25.248 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq ' 2020-03-11T22:45:25.248 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 2020-03-11T22:45:25.248 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:45:25.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705683 -lt 2020-03-11T22:45:25.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh: line 2079: test: 38654705683: unary operator expected 2020-03-11T22:45:25.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:45:25.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-184683593731 2020-03-11T22:45:25.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:45:25.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:45:25.598 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-184683593731 2020-03-11T22:45:25.598 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:45:25.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=184683593731 2020-03-11T22:45:25.599 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 184683593731' 2020-03-11T22:45:25.599 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 184683593731 2020-03-11T22:45:25.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:45:25.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 184683593731 -lt 184683593731 2020-03-11T22:45:25.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:45:25.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:25.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:26.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:45:26.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:26.443 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:26.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:26.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:26.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:26.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:26.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:26.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:26.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:26.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:26.795 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:27.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:27.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:45:27.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:45:27.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:45:27.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:27.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:45:27.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:27.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:27.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:27.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:27.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:27.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:27.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:27.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:27.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:27.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:27.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:27.667 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:28.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:28.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:28.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:28.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:28.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:28.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:28.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:28.108 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:28.108 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:28.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:28.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:28.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 1 >= 8 )) 2020-03-11T22:45:28.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:28.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.2 2020-03-11T22:45:28.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:28.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:28.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:28.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:28.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:28.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:28.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:28.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:29.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:29.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:29.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:29.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:29.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:29.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:29.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:29.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:29.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:29.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:29.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:29.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:29.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:30.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:30.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:30.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 2 >= 8 )) 2020-03-11T22:45:30.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:30.011 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.4 2020-03-11T22:45:30.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:30.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:30.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:30.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:30.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:30.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:30.412 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:30.413 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:30.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:30.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:30.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:30.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:31.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:31.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:31.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:31.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:31.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:31.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:31.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:31.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:31.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:31.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:31.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:31.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 3 >= 8 )) 2020-03-11T22:45:31.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:31.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.8 2020-03-11T22:45:32.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:32.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:32.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:32.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:32.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:32.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:32.442 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:32.442 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:32.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:32.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:32.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:32.776 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:33.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:33.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:33.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:33.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:33.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:33.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:33.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:33.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:33.224 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:33.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:33.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:33.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 4 >= 8 )) 2020-03-11T22:45:33.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:33.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 1.6 2020-03-11T22:45:35.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:35.262 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:35.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:35.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:35.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:35.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:35.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:35.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:35.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:35.604 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:35.604 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:35.604 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:36.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:36.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:36.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:36.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:36.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:36.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:36.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:36.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:36.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:36.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:36.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:36.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 5 >= 8 )) 2020-03-11T22:45:36.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:36.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 3.2 2020-03-11T22:45:39.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:39.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:39.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:39.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:39.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:39.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:39.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:39.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:40.035 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:40.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:40.036 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:40.036 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:40.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:40.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:40.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:40.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:40.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:40.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:40.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:40.486 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:40.486 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:40.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:40.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:40.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 6 >= 8 )) 2020-03-11T22:45:40.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:40.930 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 6.4 2020-03-11T22:45:47.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:47.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:47.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:47.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:47.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:47.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:47.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:47.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:47.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:47.673 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:47.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:47.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:48.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:48.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:48.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:48.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:48.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:48.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:48.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:48.115 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:48.115 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:48.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:48.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:48.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 7 >= 8 )) 2020-03-11T22:45:48.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:45:48.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 7.3 2020-03-11T22:45:55.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:45:55.866 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:45:55.867 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:45:55.867 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:45:55.867 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:45:55.867 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:45:55.867 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:45:55.868 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:45:56.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:45:56.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:45:56.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:45:56.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:45:56.651 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:45:56.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:45:56.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:45:56.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:45:56.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:45:56.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:45:56.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:45:56.653 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:45:56.653 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:45:57.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=null 2020-03-11T22:45:57.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test null '!=' null 2020-03-11T22:45:57.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1601: wait_for_clean: (( 8 >= 8 )) 2020-03-11T22:45:57.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1602: wait_for_clean: ceph report 2020-03-11T22:45:57.542 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:45:57.542 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_fingerprint": "2b255bec-2463-4ae3-b895-035b99acf8bd", 2020-03-11T22:45:57.543 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "15.1.0-1941-g4acc112", 2020-03-11T22:45:57.543 INFO:tasks.workunit.client.0.smithi155.stdout: "commit": "4acc1121e47039a3cdf2978ae458c10d153aa141", 2020-03-11T22:45:57.543 INFO:tasks.workunit.client.0.smithi155.stdout: "timestamp": "2020-03-11T22:45:57.538835+0000", 2020-03-11T22:45:57.543 INFO:tasks.workunit.client.0.smithi155.stdout: "tag": "", 2020-03-11T22:45:57.544 INFO:tasks.workunit.client.0.smithi155.stdout: "health": { 2020-03-11T22:45:57.544 INFO:tasks.workunit.client.0.smithi155.stdout: "status": "HEALTH_WARN", 2020-03-11T22:45:57.544 INFO:tasks.workunit.client.0.smithi155.stdout: "checks": { 2020-03-11T22:45:57.544 INFO:tasks.workunit.client.0.smithi155.stdout: "OSDMAP_FLAGS": { 2020-03-11T22:45:57.544 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:57.545 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:45:57.545 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "nodown,noout,noin flag(s) set", 2020-03-11T22:45:57.545 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 17 2020-03-11T22:45:57.545 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.546 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [], 2020-03-11T22:45:57.546 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:45:57.546 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.546 INFO:tasks.workunit.client.0.smithi155.stdout: "OSD_DOWN": { 2020-03-11T22:45:57.546 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:57.547 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:45:57.547 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 osds down", 2020-03-11T22:45:57.547 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:45:57.547 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.548 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:45:57.548 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.548 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "osd.1 (root=default,host=smithi155) is down" 2020-03-11T22:45:57.548 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.549 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.549 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:45:57.549 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.549 INFO:tasks.workunit.client.0.smithi155.stdout: "PG_DEGRADED": { 2020-03-11T22:45:57.549 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:57.550 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:45:57.550 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "Degraded data redundancy: 101/303 objects degraded (33.333%), 1 pg degraded", 2020-03-11T22:45:57.550 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:45:57.550 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.551 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:45:57.551 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.552 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "pg 1.0 is active+undersized+degraded, acting [2147483647,0,2]" 2020-03-11T22:45:57.552 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.552 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.552 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:45:57.553 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.553 INFO:tasks.workunit.client.0.smithi155.stdout: "POOL_APP_NOT_ENABLED": { 2020-03-11T22:45:57.553 INFO:tasks.workunit.client.0.smithi155.stdout: "severity": "HEALTH_WARN", 2020-03-11T22:45:57.553 INFO:tasks.workunit.client.0.smithi155.stdout: "summary": { 2020-03-11T22:45:57.554 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "1 pool(s) do not have an application enabled", 2020-03-11T22:45:57.554 INFO:tasks.workunit.client.0.smithi155.stdout: "count": 1 2020-03-11T22:45:57.554 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.554 INFO:tasks.workunit.client.0.smithi155.stdout: "detail": [ 2020-03-11T22:45:57.554 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.555 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "application not enabled on pool 'test'" 2020-03-11T22:45:57.555 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.555 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.555 INFO:tasks.workunit.client.0.smithi155.stdout: "message": "use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications." 2020-03-11T22:45:57.555 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.558 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.558 INFO:tasks.workunit.client.0.smithi155.stdout: "muted": false 2020-03-11T22:45:57.558 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.558 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.558 INFO:tasks.workunit.client.0.smithi155.stdout: "mutes": [] 2020-03-11T22:45:57.559 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.559 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_first_committed": 1, 2020-03-11T22:45:57.559 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap_last_committed": 1, 2020-03-11T22:45:57.559 INFO:tasks.workunit.client.0.smithi155.stdout: "monmap": { 2020-03-11T22:45:57.559 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:45:57.560 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:45:57.560 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:43:29.340201Z", 2020-03-11T22:45:57.560 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:43:29.340201Z", 2020-03-11T22:45:57.560 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release": 15, 2020-03-11T22:45:57.560 INFO:tasks.workunit.client.0.smithi155.stdout: "min_mon_release_name": "octopus", 2020-03-11T22:45:57.561 INFO:tasks.workunit.client.0.smithi155.stdout: "features": { 2020-03-11T22:45:57.561 INFO:tasks.workunit.client.0.smithi155.stdout: "persistent": [ 2020-03-11T22:45:57.561 INFO:tasks.workunit.client.0.smithi155.stdout: "kraken", 2020-03-11T22:45:57.561 INFO:tasks.workunit.client.0.smithi155.stdout: "luminous", 2020-03-11T22:45:57.561 INFO:tasks.workunit.client.0.smithi155.stdout: "mimic", 2020-03-11T22:45:57.561 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap-prune", 2020-03-11T22:45:57.561 INFO:tasks.workunit.client.0.smithi155.stdout: "nautilus", 2020-03-11T22:45:57.562 INFO:tasks.workunit.client.0.smithi155.stdout: "octopus" 2020-03-11T22:45:57.562 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.562 INFO:tasks.workunit.client.0.smithi155.stdout: "optional": [] 2020-03-11T22:45:57.562 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.562 INFO:tasks.workunit.client.0.smithi155.stdout: "mons": [ 2020-03-11T22:45:57.562 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.562 INFO:tasks.workunit.client.0.smithi155.stdout: "rank": 0, 2020-03-11T22:45:57.563 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "a", 2020-03-11T22:45:57.563 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:45:57.563 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.563 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.563 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.563 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115", 2020-03-11T22:45:57.563 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 0 2020-03-11T22:45:57.564 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.564 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.564 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.564 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:7115/0", 2020-03-11T22:45:57.564 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:7115/0", 2020-03-11T22:45:57.564 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 0, 2020-03-11T22:45:57.564 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 0 2020-03-11T22:45:57.565 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.565 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.565 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.565 INFO:tasks.workunit.client.0.smithi155.stdout: "quorum": [ 2020-03-11T22:45:57.565 INFO:tasks.workunit.client.0.smithi155.stdout: 0 2020-03-11T22:45:57.565 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.565 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap": { 2020-03-11T22:45:57.566 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 44, 2020-03-11T22:45:57.566 INFO:tasks.workunit.client.0.smithi155.stdout: "fsid": "02364139-af2c-42f4-b33a-c83055e85709", 2020-03-11T22:45:57.566 INFO:tasks.workunit.client.0.smithi155.stdout: "created": "2020-03-11T22:43:30.109657+0000", 2020-03-11T22:45:57.566 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:45:22.812541+0000", 2020-03-11T22:45:57.566 INFO:tasks.workunit.client.0.smithi155.stdout: "last_up_change": "2020-03-11T22:45:22.640821+0000", 2020-03-11T22:45:57.566 INFO:tasks.workunit.client.0.smithi155.stdout: "last_in_change": "2020-03-11T22:44:37.061242+0000", 2020-03-11T22:45:57.567 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": "nodown,noout,noin,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2020-03-11T22:45:57.567 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_num": 5800384, 2020-03-11T22:45:57.567 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_set": [ 2020-03-11T22:45:57.567 INFO:tasks.workunit.client.0.smithi155.stdout: "nodown", 2020-03-11T22:45:57.567 INFO:tasks.workunit.client.0.smithi155.stdout: "noin", 2020-03-11T22:45:57.567 INFO:tasks.workunit.client.0.smithi155.stdout: "noout", 2020-03-11T22:45:57.567 INFO:tasks.workunit.client.0.smithi155.stdout: "pglog_hardlimit", 2020-03-11T22:45:57.568 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snapdirs", 2020-03-11T22:45:57.568 INFO:tasks.workunit.client.0.smithi155.stdout: "recovery_deletes", 2020-03-11T22:45:57.568 INFO:tasks.workunit.client.0.smithi155.stdout: "sortbitwise" 2020-03-11T22:45:57.568 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.568 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_version": 8, 2020-03-11T22:45:57.568 INFO:tasks.workunit.client.0.smithi155.stdout: "full_ratio": 0.99000000953674316, 2020-03-11T22:45:57.568 INFO:tasks.workunit.client.0.smithi155.stdout: "backfillfull_ratio": 0.99000000953674316, 2020-03-11T22:45:57.568 INFO:tasks.workunit.client.0.smithi155.stdout: "nearfull_ratio": 0.85000002384185791, 2020-03-11T22:45:57.569 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_snapshot": "", 2020-03-11T22:45:57.569 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_max": 1, 2020-03-11T22:45:57.569 INFO:tasks.workunit.client.0.smithi155.stdout: "max_osd": 3, 2020-03-11T22:45:57.569 INFO:tasks.workunit.client.0.smithi155.stdout: "require_min_compat_client": "jewel", 2020-03-11T22:45:57.569 INFO:tasks.workunit.client.0.smithi155.stdout: "min_compat_client": "jewel", 2020-03-11T22:45:57.569 INFO:tasks.workunit.client.0.smithi155.stdout: "require_osd_release": "octopus", 2020-03-11T22:45:57.570 INFO:tasks.workunit.client.0.smithi155.stdout: "pools": [ 2020-03-11T22:45:57.570 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.570 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:45:57.570 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_name": "test", 2020-03-11T22:45:57.570 INFO:tasks.workunit.client.0.smithi155.stdout: "create_time": "2020-03-11T22:43:52.806296+0000", 2020-03-11T22:45:57.570 INFO:tasks.workunit.client.0.smithi155.stdout: "flags": 16389, 2020-03-11T22:45:57.571 INFO:tasks.workunit.client.0.smithi155.stdout: "flags_names": "hashpspool,ec_overwrites,pool_snaps", 2020-03-11T22:45:57.571 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 3, 2020-03-11T22:45:57.571 INFO:tasks.workunit.client.0.smithi155.stdout: "size": 3, 2020-03-11T22:45:57.571 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 2, 2020-03-11T22:45:57.571 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_rule": 1, 2020-03-11T22:45:57.571 INFO:tasks.workunit.client.0.smithi155.stdout: "object_hash": 2, 2020-03-11T22:45:57.571 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_autoscale_mode": "off", 2020-03-11T22:45:57.572 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num": 1, 2020-03-11T22:45:57.572 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num": 1, 2020-03-11T22:45:57.572 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_placement_num_target": 1, 2020-03-11T22:45:57.572 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_target": 1, 2020-03-11T22:45:57.572 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_num_pending": 1, 2020-03-11T22:45:57.572 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pg_merge_meta": { 2020-03-11T22:45:57.572 INFO:tasks.workunit.client.0.smithi155.stdout: "source_pgid": "0.0", 2020-03-11T22:45:57.573 INFO:tasks.workunit.client.0.smithi155.stdout: "ready_epoch": 0, 2020-03-11T22:45:57.573 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_started": 0, 2020-03-11T22:45:57.573 INFO:tasks.workunit.client.0.smithi155.stdout: "last_epoch_clean": 0, 2020-03-11T22:45:57.573 INFO:tasks.workunit.client.0.smithi155.stdout: "source_version": "0'0", 2020-03-11T22:45:57.573 INFO:tasks.workunit.client.0.smithi155.stdout: "target_version": "0'0" 2020-03-11T22:45:57.573 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.573 INFO:tasks.workunit.client.0.smithi155.stdout: "last_change": "28", 2020-03-11T22:45:57.574 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend": "0", 2020-03-11T22:45:57.574 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_prenautilus": "0", 2020-03-11T22:45:57.574 INFO:tasks.workunit.client.0.smithi155.stdout: "last_force_op_resend_preluminous": "0", 2020-03-11T22:45:57.574 INFO:tasks.workunit.client.0.smithi155.stdout: "auid": 0, 2020-03-11T22:45:57.574 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_mode": "pool", 2020-03-11T22:45:57.574 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_seq": 1, 2020-03-11T22:45:57.574 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_epoch": 28, 2020-03-11T22:45:57.575 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_snaps": [ 2020-03-11T22:45:57.575 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.575 INFO:tasks.workunit.client.0.smithi155.stdout: "snapid": 1, 2020-03-11T22:45:57.575 INFO:tasks.workunit.client.0.smithi155.stdout: "stamp": "2020-03-11T22:44:18.508398+0000", 2020-03-11T22:45:57.575 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "snap1" 2020-03-11T22:45:57.575 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.575 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.575 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps": "[]", 2020-03-11T22:45:57.576 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_bytes": 0, 2020-03-11T22:45:57.576 INFO:tasks.workunit.client.0.smithi155.stdout: "quota_max_objects": 0, 2020-03-11T22:45:57.576 INFO:tasks.workunit.client.0.smithi155.stdout: "tiers": [], 2020-03-11T22:45:57.576 INFO:tasks.workunit.client.0.smithi155.stdout: "tier_of": -1, 2020-03-11T22:45:57.576 INFO:tasks.workunit.client.0.smithi155.stdout: "read_tier": -1, 2020-03-11T22:45:57.576 INFO:tasks.workunit.client.0.smithi155.stdout: "write_tier": -1, 2020-03-11T22:45:57.576 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_mode": "none", 2020-03-11T22:45:57.577 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_bytes": 0, 2020-03-11T22:45:57.577 INFO:tasks.workunit.client.0.smithi155.stdout: "target_max_objects": 0, 2020-03-11T22:45:57.577 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_ratio_micro": 400000, 2020-03-11T22:45:57.577 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2020-03-11T22:45:57.577 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_target_full_ratio_micro": 800000, 2020-03-11T22:45:57.577 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_flush_age": 0, 2020-03-11T22:45:57.577 INFO:tasks.workunit.client.0.smithi155.stdout: "cache_min_evict_age": 0, 2020-03-11T22:45:57.578 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profile": "myprofile", 2020-03-11T22:45:57.578 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_params": { 2020-03-11T22:45:57.578 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "none" 2020-03-11T22:45:57.578 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.578 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_period": 0, 2020-03-11T22:45:57.578 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_count": 0, 2020-03-11T22:45:57.578 INFO:tasks.workunit.client.0.smithi155.stdout: "use_gmt_hitset": true, 2020-03-11T22:45:57.578 INFO:tasks.workunit.client.0.smithi155.stdout: "min_read_recency_for_promote": 0, 2020-03-11T22:45:57.579 INFO:tasks.workunit.client.0.smithi155.stdout: "min_write_recency_for_promote": 0, 2020-03-11T22:45:57.579 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_grade_decay_rate": 0, 2020-03-11T22:45:57.579 INFO:tasks.workunit.client.0.smithi155.stdout: "hit_set_search_last_n": 0, 2020-03-11T22:45:57.579 INFO:tasks.workunit.client.0.smithi155.stdout: "grade_table": [], 2020-03-11T22:45:57.579 INFO:tasks.workunit.client.0.smithi155.stdout: "stripe_width": 8192, 2020-03-11T22:45:57.579 INFO:tasks.workunit.client.0.smithi155.stdout: "expected_num_objects": 0, 2020-03-11T22:45:57.579 INFO:tasks.workunit.client.0.smithi155.stdout: "fast_read": false, 2020-03-11T22:45:57.580 INFO:tasks.workunit.client.0.smithi155.stdout: "options": {}, 2020-03-11T22:45:57.580 INFO:tasks.workunit.client.0.smithi155.stdout: "application_metadata": {} 2020-03-11T22:45:57.580 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.580 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.580 INFO:tasks.workunit.client.0.smithi155.stdout: "osds": [ 2020-03-11T22:45:57.580 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.580 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:45:57.580 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "e0e5724e-7440-465c-9db6-82e8c737bcc8", 2020-03-11T22:45:57.581 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:45:57.581 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:45:57.581 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:45:57.581 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:45:57.581 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 34, 2020-03-11T22:45:57.581 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 38, 2020-03-11T22:45:57.581 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 40, 2020-03-11T22:45:57.582 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 43, 2020-03-11T22:45:57.582 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 39, 2020-03-11T22:45:57.582 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:45:57.582 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:45:57.582 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.582 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.582 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.583 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6802", 2020-03-11T22:45:57.583 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50341 2020-03-11T22:45:57.583 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.583 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.583 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:57.583 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6803", 2020-03-11T22:45:57.583 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50341 2020-03-11T22:45:57.584 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.584 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.584 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.584 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:45:57.584 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.584 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.584 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.584 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6804", 2020-03-11T22:45:57.585 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50341 2020-03-11T22:45:57.585 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.585 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.585 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:57.585 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6805", 2020-03-11T22:45:57.585 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50341 2020-03-11T22:45:57.585 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.586 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.586 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.586 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:45:57.586 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.586 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.586 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.586 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6808", 2020-03-11T22:45:57.586 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50341 2020-03-11T22:45:57.587 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.587 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.587 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:57.587 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6809", 2020-03-11T22:45:57.587 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50341 2020-03-11T22:45:57.587 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.587 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.587 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.588 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:45:57.588 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.588 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.588 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.588 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6806", 2020-03-11T22:45:57.588 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50341 2020-03-11T22:45:57.588 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.589 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.589 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:57.589 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6807", 2020-03-11T22:45:57.589 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50341 2020-03-11T22:45:57.589 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.589 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.589 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.590 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6803/50341", 2020-03-11T22:45:57.590 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6805/50341", 2020-03-11T22:45:57.590 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6809/50341", 2020-03-11T22:45:57.590 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6807/50341", 2020-03-11T22:45:57.590 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:45:57.590 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:45:57.591 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:45:57.591 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.591 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.591 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.591 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:45:57.591 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "7b6947d8-603c-41a4-abcb-8b50325a2007", 2020-03-11T22:45:57.591 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 0, 2020-03-11T22:45:57.592 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:45:57.592 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:45:57.592 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:45:57.592 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 0, 2020-03-11T22:45:57.592 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 0, 2020-03-11T22:45:57.592 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 9, 2020-03-11T22:45:57.592 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 23, 2020-03-11T22:45:57.593 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 30, 2020-03-11T22:45:57.593 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:45:57.593 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:45:57.593 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.593 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.593 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.593 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:45:57.593 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:57.594 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.594 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.594 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:57.594 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:45:57.594 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:57.594 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.594 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.594 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.595 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:45:57.595 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.595 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.595 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.595 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:45:57.595 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:57.595 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.596 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.596 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:57.596 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:45:57.596 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:57.596 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.596 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.596 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.596 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:45:57.597 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.597 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.597 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.597 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:45:57.597 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:57.597 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.597 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.598 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:57.598 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:45:57.598 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:57.598 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.598 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.598 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.598 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:45:57.599 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.599 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.599 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.599 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:45:57.599 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:57.599 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.599 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.600 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:57.600 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:45:57.600 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 43970 2020-03-11T22:45:57.600 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.600 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.600 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.600 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/43970", 2020-03-11T22:45:57.601 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/43970", 2020-03-11T22:45:57.601 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/43970", 2020-03-11T22:45:57.601 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/43970", 2020-03-11T22:45:57.601 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:45:57.601 INFO:tasks.workunit.client.0.smithi155.stdout: "exists" 2020-03-11T22:45:57.602 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.602 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.602 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.602 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:45:57.602 INFO:tasks.workunit.client.0.smithi155.stdout: "uuid": "797c47f4-417e-4663-8ca9-e4cc78330dd9", 2020-03-11T22:45:57.602 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 1, 2020-03-11T22:45:57.602 INFO:tasks.workunit.client.0.smithi155.stdout: "in": 1, 2020-03-11T22:45:57.602 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 1, 2020-03-11T22:45:57.603 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_affinity": 1, 2020-03-11T22:45:57.603 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_begin": 35, 2020-03-11T22:45:57.603 INFO:tasks.workunit.client.0.smithi155.stdout: "last_clean_end": 38, 2020-03-11T22:45:57.603 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 43, 2020-03-11T22:45:57.603 INFO:tasks.workunit.client.0.smithi155.stdout: "up_thru": 0, 2020-03-11T22:45:57.603 INFO:tasks.workunit.client.0.smithi155.stdout: "down_at": 42, 2020-03-11T22:45:57.603 INFO:tasks.workunit.client.0.smithi155.stdout: "lost_at": 0, 2020-03-11T22:45:57.604 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addrs": { 2020-03-11T22:45:57.604 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.604 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.604 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.604 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6810", 2020-03-11T22:45:57.604 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50581 2020-03-11T22:45:57.605 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.605 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.605 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:57.605 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6811", 2020-03-11T22:45:57.606 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50581 2020-03-11T22:45:57.606 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.606 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.606 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.606 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addrs": { 2020-03-11T22:45:57.606 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.606 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.606 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.607 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6812", 2020-03-11T22:45:57.607 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50581 2020-03-11T22:45:57.607 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.607 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.607 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:57.608 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6813", 2020-03-11T22:45:57.608 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50581 2020-03-11T22:45:57.608 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.608 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.608 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.608 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addrs": { 2020-03-11T22:45:57.608 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.609 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.609 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.609 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6816", 2020-03-11T22:45:57.609 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50581 2020-03-11T22:45:57.609 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.609 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.609 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:57.610 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6817", 2020-03-11T22:45:57.610 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50581 2020-03-11T22:45:57.610 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.610 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.610 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.610 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addrs": { 2020-03-11T22:45:57.610 INFO:tasks.workunit.client.0.smithi155.stdout: "addrvec": [ 2020-03-11T22:45:57.610 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.611 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v2", 2020-03-11T22:45:57.611 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6814", 2020-03-11T22:45:57.611 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50581 2020-03-11T22:45:57.611 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.611 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.611 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "v1", 2020-03-11T22:45:57.611 INFO:tasks.workunit.client.0.smithi155.stdout: "addr": "127.0.0.1:6815", 2020-03-11T22:45:57.612 INFO:tasks.workunit.client.0.smithi155.stdout: "nonce": 50581 2020-03-11T22:45:57.612 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.612 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.612 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.612 INFO:tasks.workunit.client.0.smithi155.stdout: "public_addr": "127.0.0.1:6811/50581", 2020-03-11T22:45:57.612 INFO:tasks.workunit.client.0.smithi155.stdout: "cluster_addr": "127.0.0.1:6813/50581", 2020-03-11T22:45:57.612 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_back_addr": "127.0.0.1:6817/50581", 2020-03-11T22:45:57.612 INFO:tasks.workunit.client.0.smithi155.stdout: "heartbeat_front_addr": "127.0.0.1:6815/50581", 2020-03-11T22:45:57.613 INFO:tasks.workunit.client.0.smithi155.stdout: "state": [ 2020-03-11T22:45:57.613 INFO:tasks.workunit.client.0.smithi155.stdout: "exists", 2020-03-11T22:45:57.613 INFO:tasks.workunit.client.0.smithi155.stdout: "up" 2020-03-11T22:45:57.613 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.613 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.613 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.613 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_xinfo": [ 2020-03-11T22:45:57.613 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.614 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:45:57.614 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:45:20.369120+0000", 2020-03-11T22:45:57.614 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:45:57.614 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:45:57.614 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:45:57.614 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:45:57.614 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:37.264836+0000", 2020-03-11T22:45:57.615 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:45:57.615 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.615 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.615 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:45:57.615 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:44:29.934725+0000", 2020-03-11T22:45:57.615 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:45:57.615 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:45:57.616 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:45:57.616 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:45:57.616 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:43.587931+0000", 2020-03-11T22:45:57.616 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:45:57.616 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.616 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.616 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:45:57.616 INFO:tasks.workunit.client.0.smithi155.stdout: "down_stamp": "2020-03-11T22:45:22.465573+0000", 2020-03-11T22:45:57.617 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_probability": 0, 2020-03-11T22:45:57.617 INFO:tasks.workunit.client.0.smithi155.stdout: "laggy_interval": 0, 2020-03-11T22:45:57.617 INFO:tasks.workunit.client.0.smithi155.stdout: "features": 4540138292836696063, 2020-03-11T22:45:57.617 INFO:tasks.workunit.client.0.smithi155.stdout: "old_weight": 0, 2020-03-11T22:45:57.617 INFO:tasks.workunit.client.0.smithi155.stdout: "last_purged_snaps_scrub": "2020-03-11T22:43:49.866136+0000", 2020-03-11T22:45:57.617 INFO:tasks.workunit.client.0.smithi155.stdout: "dead_epoch": 0 2020-03-11T22:45:57.617 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.618 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.618 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap": [], 2020-03-11T22:45:57.618 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_upmap_items": [], 2020-03-11T22:45:57.618 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_temp": [], 2020-03-11T22:45:57.618 INFO:tasks.workunit.client.0.smithi155.stdout: "primary_temp": [], 2020-03-11T22:45:57.618 INFO:tasks.workunit.client.0.smithi155.stdout: "blacklist": {}, 2020-03-11T22:45:57.618 INFO:tasks.workunit.client.0.smithi155.stdout: "erasure_code_profiles": { 2020-03-11T22:45:57.618 INFO:tasks.workunit.client.0.smithi155.stdout: "default": { 2020-03-11T22:45:57.619 INFO:tasks.workunit.client.0.smithi155.stdout: "k": "2", 2020-03-11T22:45:57.619 INFO:tasks.workunit.client.0.smithi155.stdout: "m": "2", 2020-03-11T22:45:57.619 INFO:tasks.workunit.client.0.smithi155.stdout: "plugin": "jerasure", 2020-03-11T22:45:57.619 INFO:tasks.workunit.client.0.smithi155.stdout: "technique": "reed_sol_van" 2020-03-11T22:45:57.619 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.619 INFO:tasks.workunit.client.0.smithi155.stdout: "myprofile": { 2020-03-11T22:45:57.619 INFO:tasks.workunit.client.0.smithi155.stdout: "crush-device-class": "", 2020-03-11T22:45:57.620 INFO:tasks.workunit.client.0.smithi155.stdout: "crush-failure-domain": "osd", 2020-03-11T22:45:57.620 INFO:tasks.workunit.client.0.smithi155.stdout: "crush-root": "default", 2020-03-11T22:45:57.620 INFO:tasks.workunit.client.0.smithi155.stdout: "jerasure-per-chunk-alignment": "false", 2020-03-11T22:45:57.620 INFO:tasks.workunit.client.0.smithi155.stdout: "k": "2", 2020-03-11T22:45:57.620 INFO:tasks.workunit.client.0.smithi155.stdout: "m": "1", 2020-03-11T22:45:57.620 INFO:tasks.workunit.client.0.smithi155.stdout: "plugin": "jerasure", 2020-03-11T22:45:57.620 INFO:tasks.workunit.client.0.smithi155.stdout: "technique": "reed_sol_van", 2020-03-11T22:45:57.620 INFO:tasks.workunit.client.0.smithi155.stdout: "w": "8" 2020-03-11T22:45:57.621 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.621 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.621 INFO:tasks.workunit.client.0.smithi155.stdout: "removed_snaps_queue": [], 2020-03-11T22:45:57.621 INFO:tasks.workunit.client.0.smithi155.stdout: "new_removed_snaps": [], 2020-03-11T22:45:57.621 INFO:tasks.workunit.client.0.smithi155.stdout: "new_purged_snaps": [], 2020-03-11T22:45:57.621 INFO:tasks.workunit.client.0.smithi155.stdout: "crush_node_flags": {}, 2020-03-11T22:45:57.621 INFO:tasks.workunit.client.0.smithi155.stdout: "device_class_flags": {} 2020-03-11T22:45:57.622 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.622 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_metadata": [ 2020-03-11T22:45:57.622 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.622 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:45:57.622 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:45:57.622 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6804/50341,v1:127.0.0.1:6805/50341]", 2020-03-11T22:45:57.622 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:45:57.622 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:45:57.623 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:45:57.623 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:45:57.623 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:45:57.623 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:45:57.623 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:45:57.623 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:45:57.623 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:45:57.623 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/0/block", 2020-03-11T22:45:57.624 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:45:57.624 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:45:57.624 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:45:57.624 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:45:57.624 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:45:57.624 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:45:57.624 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:45:57.625 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:45:57.625 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:45:57.625 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:45:57.625 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:45:57.625 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:45:57.625 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:45:57.625 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:45:57.626 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:45:57.626 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6802/50341,v1:127.0.0.1:6803/50341]", 2020-03-11T22:45:57.626 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:45:57.626 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6808/50341,v1:127.0.0.1:6809/50341]", 2020-03-11T22:45:57.626 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6806/50341,v1:127.0.0.1:6807/50341]", 2020-03-11T22:45:57.626 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:45:57.626 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:45:57.626 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:45:57.627 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:45:57.627 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:45:57.627 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:45:57.627 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:45:57.627 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:45:57.627 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:45:57.627 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/0", 2020-03-11T22:45:57.627 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:45:57.628 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:45:57.628 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.628 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.628 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:45:57.628 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:45:57.628 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/43970,v1:127.0.0.1:6813/43970]", 2020-03-11T22:45:57.628 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:45:57.629 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:45:57.629 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:45:57.629 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:45:57.629 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:45:57.629 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:45:57.629 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:45:57.629 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:45:57.629 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:45:57.630 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/1/block", 2020-03-11T22:45:57.630 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:45:57.630 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:45:57.630 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:45:57.630 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:45:57.630 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:45:57.630 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:45:57.631 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:45:57.631 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:45:57.631 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:45:57.631 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:45:57.631 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:45:57.631 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:45:57.631 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:45:57.631 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:45:57.632 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:45:57.632 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/43970,v1:127.0.0.1:6811/43970]", 2020-03-11T22:45:57.632 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:45:57.632 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/43970,v1:127.0.0.1:6817/43970]", 2020-03-11T22:45:57.632 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/43970,v1:127.0.0.1:6815/43970]", 2020-03-11T22:45:57.632 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:45:57.632 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:45:57.633 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:45:57.633 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:45:57.633 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:45:57.633 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:45:57.633 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:45:57.633 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:45:57.633 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:45:57.633 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/1", 2020-03-11T22:45:57.634 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:45:57.634 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:45:57.634 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.634 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.634 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:45:57.634 INFO:tasks.workunit.client.0.smithi155.stdout: "arch": "x86_64", 2020-03-11T22:45:57.634 INFO:tasks.workunit.client.0.smithi155.stdout: "back_addr": "[v2:127.0.0.1:6812/50581,v1:127.0.0.1:6813/50581]", 2020-03-11T22:45:57.635 INFO:tasks.workunit.client.0.smithi155.stdout: "back_iface": "", 2020-03-11T22:45:57.635 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs": "1", 2020-03-11T22:45:57.635 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_db": "0", 2020-03-11T22:45:57.635 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_dedicated_wal": "0", 2020-03-11T22:45:57.635 INFO:tasks.workunit.client.0.smithi155.stdout: "bluefs_single_shared_device": "1", 2020-03-11T22:45:57.635 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_access_mode": "file", 2020-03-11T22:45:57.635 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_block_size": "4096", 2020-03-11T22:45:57.635 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_devices": "sda", 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_driver": "KernelDevice", 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_path": "td/divergent-priors/2/block", 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_rotational": "1", 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_size": "107374182400", 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_support_discard": "0", 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi155.stdout: "bluestore_bdev_type": "hdd", 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_release": "octopus", 2020-03-11T22:45:57.636 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version": "ceph version 15.1.0-1941-g4acc112 (4acc1121e47039a3cdf2978ae458c10d153aa141) octopus (rc)", 2020-03-11T22:45:57.637 INFO:tasks.workunit.client.0.smithi155.stdout: "ceph_version_short": "15.1.0-1941-g4acc112", 2020-03-11T22:45:57.637 INFO:tasks.workunit.client.0.smithi155.stdout: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2020-03-11T22:45:57.637 INFO:tasks.workunit.client.0.smithi155.stdout: "default_device_class": "hdd", 2020-03-11T22:45:57.637 INFO:tasks.workunit.client.0.smithi155.stdout: "device_ids": "sda=ST1000NM0033-9ZM173_Z1W5XNRW", 2020-03-11T22:45:57.637 INFO:tasks.workunit.client.0.smithi155.stdout: "device_paths": "sda=/dev/disk/by-path/pci-0000:00:1f.2-ata-1", 2020-03-11T22:45:57.637 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": "sda", 2020-03-11T22:45:57.637 INFO:tasks.workunit.client.0.smithi155.stdout: "distro": "centos", 2020-03-11T22:45:57.638 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_description": "CentOS Linux 8 (Core)", 2020-03-11T22:45:57.638 INFO:tasks.workunit.client.0.smithi155.stdout: "distro_version": "8", 2020-03-11T22:45:57.638 INFO:tasks.workunit.client.0.smithi155.stdout: "front_addr": "[v2:127.0.0.1:6810/50581,v1:127.0.0.1:6811/50581]", 2020-03-11T22:45:57.638 INFO:tasks.workunit.client.0.smithi155.stdout: "front_iface": "", 2020-03-11T22:45:57.638 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_back_addr": "[v2:127.0.0.1:6816/50581,v1:127.0.0.1:6817/50581]", 2020-03-11T22:45:57.638 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_front_addr": "[v2:127.0.0.1:6814/50581,v1:127.0.0.1:6815/50581]", 2020-03-11T22:45:57.638 INFO:tasks.workunit.client.0.smithi155.stdout: "hostname": "smithi155", 2020-03-11T22:45:57.639 INFO:tasks.workunit.client.0.smithi155.stdout: "journal_rotational": "1", 2020-03-11T22:45:57.639 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_description": "#1 SMP Wed Dec 4 21:51:45 UTC 2019", 2020-03-11T22:45:57.639 INFO:tasks.workunit.client.0.smithi155.stdout: "kernel_version": "4.18.0-147.el8.x86_64", 2020-03-11T22:45:57.639 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_swap_kb": "0", 2020-03-11T22:45:57.639 INFO:tasks.workunit.client.0.smithi155.stdout: "mem_total_kb": "32655912", 2020-03-11T22:45:57.639 INFO:tasks.workunit.client.0.smithi155.stdout: "network_numa_unknown_ifaces": "back_iface,front_iface", 2020-03-11T22:45:57.639 INFO:tasks.workunit.client.0.smithi155.stdout: "objectstore_numa_unknown_devices": "sda", 2020-03-11T22:45:57.639 INFO:tasks.workunit.client.0.smithi155.stdout: "os": "Linux", 2020-03-11T22:45:57.640 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_data": "td/divergent-priors/2", 2020-03-11T22:45:57.640 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_objectstore": "bluestore", 2020-03-11T22:45:57.640 INFO:tasks.workunit.client.0.smithi155.stdout: "rotational": "1" 2020-03-11T22:45:57.640 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.640 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.640 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_first_committed": 1, 2020-03-11T22:45:57.640 INFO:tasks.workunit.client.0.smithi155.stdout: "osdmap_last_committed": 44, 2020-03-11T22:45:57.640 INFO:tasks.workunit.client.0.smithi155.stdout: "crushmap": { 2020-03-11T22:45:57.641 INFO:tasks.workunit.client.0.smithi155.stdout: "devices": [ 2020-03-11T22:45:57.641 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.641 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:45:57.641 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.0", 2020-03-11T22:45:57.641 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:45:57.641 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.641 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.642 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:45:57.642 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.1", 2020-03-11T22:45:57.642 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:45:57.642 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.642 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.642 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:45:57.642 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd.2", 2020-03-11T22:45:57.643 INFO:tasks.workunit.client.0.smithi155.stdout: "class": "hdd" 2020-03-11T22:45:57.643 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.643 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.643 INFO:tasks.workunit.client.0.smithi155.stdout: "types": [ 2020-03-11T22:45:57.643 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.643 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 0, 2020-03-11T22:45:57.643 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "osd" 2020-03-11T22:45:57.643 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.644 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.644 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:45:57.644 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "host" 2020-03-11T22:45:57.644 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.644 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.644 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 2, 2020-03-11T22:45:57.644 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "chassis" 2020-03-11T22:45:57.645 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.645 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.645 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 3, 2020-03-11T22:45:57.645 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "rack" 2020-03-11T22:45:57.645 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.645 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.645 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 4, 2020-03-11T22:45:57.645 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "row" 2020-03-11T22:45:57.646 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.646 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.646 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 5, 2020-03-11T22:45:57.646 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pdu" 2020-03-11T22:45:57.646 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.646 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.646 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 6, 2020-03-11T22:45:57.647 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "pod" 2020-03-11T22:45:57.647 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.647 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.647 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 7, 2020-03-11T22:45:57.647 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "room" 2020-03-11T22:45:57.647 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.647 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.647 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 8, 2020-03-11T22:45:57.648 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "datacenter" 2020-03-11T22:45:57.648 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.648 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.648 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 9, 2020-03-11T22:45:57.648 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "zone" 2020-03-11T22:45:57.648 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.648 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.648 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 10, 2020-03-11T22:45:57.649 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "region" 2020-03-11T22:45:57.649 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.649 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.649 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:45:57.649 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "root" 2020-03-11T22:45:57.649 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.649 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.650 INFO:tasks.workunit.client.0.smithi155.stdout: "buckets": [ 2020-03-11T22:45:57.650 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.650 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -1, 2020-03-11T22:45:57.650 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default", 2020-03-11T22:45:57.650 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:45:57.650 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:45:57.650 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:45:57.651 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:45:57.651 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:45:57.651 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:45:57.651 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.651 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:45:57.651 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:45:57.651 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:45:57.651 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.652 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.652 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.652 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.652 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -2, 2020-03-11T22:45:57.652 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "default~hdd", 2020-03-11T22:45:57.652 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 11, 2020-03-11T22:45:57.652 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "root", 2020-03-11T22:45:57.653 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:45:57.653 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:45:57.653 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:45:57.653 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:45:57.653 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.653 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:45:57.653 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:45:57.654 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:45:57.654 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.654 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.654 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.654 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.654 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -3, 2020-03-11T22:45:57.654 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155", 2020-03-11T22:45:57.655 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:45:57.655 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:45:57.655 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:45:57.655 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:45:57.655 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:45:57.655 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:45:57.655 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.656 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:45:57.656 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:45:57.656 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:45:57.656 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.656 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.657 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:45:57.657 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:45:57.657 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:45:57.657 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.657 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.657 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:45:57.658 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:45:57.658 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:45:57.658 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.658 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.658 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.658 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.658 INFO:tasks.workunit.client.0.smithi155.stdout: "id": -4, 2020-03-11T22:45:57.658 INFO:tasks.workunit.client.0.smithi155.stdout: "name": "smithi155~hdd", 2020-03-11T22:45:57.659 INFO:tasks.workunit.client.0.smithi155.stdout: "type_id": 1, 2020-03-11T22:45:57.659 INFO:tasks.workunit.client.0.smithi155.stdout: "type_name": "host", 2020-03-11T22:45:57.659 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 19206, 2020-03-11T22:45:57.659 INFO:tasks.workunit.client.0.smithi155.stdout: "alg": "straw2", 2020-03-11T22:45:57.659 INFO:tasks.workunit.client.0.smithi155.stdout: "hash": "rjenkins1", 2020-03-11T22:45:57.659 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:45:57.659 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.660 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 0, 2020-03-11T22:45:57.660 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:45:57.660 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 0 2020-03-11T22:45:57.660 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.660 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.660 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 1, 2020-03-11T22:45:57.661 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:45:57.661 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 1 2020-03-11T22:45:57.661 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.661 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.661 INFO:tasks.workunit.client.0.smithi155.stdout: "id": 2, 2020-03-11T22:45:57.661 INFO:tasks.workunit.client.0.smithi155.stdout: "weight": 6402, 2020-03-11T22:45:57.661 INFO:tasks.workunit.client.0.smithi155.stdout: "pos": 2 2020-03-11T22:45:57.662 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.662 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.662 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.662 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.662 INFO:tasks.workunit.client.0.smithi155.stdout: "rules": [ 2020-03-11T22:45:57.662 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.662 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_id": 0, 2020-03-11T22:45:57.662 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_name": "replicated_rule", 2020-03-11T22:45:57.663 INFO:tasks.workunit.client.0.smithi155.stdout: "ruleset": 0, 2020-03-11T22:45:57.663 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 1, 2020-03-11T22:45:57.663 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 1, 2020-03-11T22:45:57.663 INFO:tasks.workunit.client.0.smithi155.stdout: "max_size": 10, 2020-03-11T22:45:57.663 INFO:tasks.workunit.client.0.smithi155.stdout: "steps": [ 2020-03-11T22:45:57.663 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.664 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "take", 2020-03-11T22:45:57.664 INFO:tasks.workunit.client.0.smithi155.stdout: "item": -1, 2020-03-11T22:45:57.664 INFO:tasks.workunit.client.0.smithi155.stdout: "item_name": "default" 2020-03-11T22:45:57.664 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.664 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.664 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "choose_firstn", 2020-03-11T22:45:57.664 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 0, 2020-03-11T22:45:57.664 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "osd" 2020-03-11T22:45:57.665 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.665 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.665 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "emit" 2020-03-11T22:45:57.665 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.665 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.665 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.665 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.666 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_id": 1, 2020-03-11T22:45:57.666 INFO:tasks.workunit.client.0.smithi155.stdout: "rule_name": "test", 2020-03-11T22:45:57.666 INFO:tasks.workunit.client.0.smithi155.stdout: "ruleset": 1, 2020-03-11T22:45:57.666 INFO:tasks.workunit.client.0.smithi155.stdout: "type": 3, 2020-03-11T22:45:57.666 INFO:tasks.workunit.client.0.smithi155.stdout: "min_size": 3, 2020-03-11T22:45:57.666 INFO:tasks.workunit.client.0.smithi155.stdout: "max_size": 3, 2020-03-11T22:45:57.666 INFO:tasks.workunit.client.0.smithi155.stdout: "steps": [ 2020-03-11T22:45:57.667 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.667 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "set_chooseleaf_tries", 2020-03-11T22:45:57.667 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 5 2020-03-11T22:45:57.667 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.667 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.667 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "set_choose_tries", 2020-03-11T22:45:57.668 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 100 2020-03-11T22:45:57.668 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.668 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.668 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "take", 2020-03-11T22:45:57.668 INFO:tasks.workunit.client.0.smithi155.stdout: "item": -1, 2020-03-11T22:45:57.668 INFO:tasks.workunit.client.0.smithi155.stdout: "item_name": "default" 2020-03-11T22:45:57.668 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.669 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.669 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "choose_indep", 2020-03-11T22:45:57.669 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 0, 2020-03-11T22:45:57.669 INFO:tasks.workunit.client.0.smithi155.stdout: "type": "osd" 2020-03-11T22:45:57.669 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.669 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.669 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "emit" 2020-03-11T22:45:57.670 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.670 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:45:57.670 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.670 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.670 INFO:tasks.workunit.client.0.smithi155.stdout: "tunables": { 2020-03-11T22:45:57.670 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_tries": 0, 2020-03-11T22:45:57.670 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_local_fallback_tries": 0, 2020-03-11T22:45:57.671 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_total_tries": 50, 2020-03-11T22:45:57.671 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_descend_once": 1, 2020-03-11T22:45:57.671 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_vary_r": 1, 2020-03-11T22:45:57.671 INFO:tasks.workunit.client.0.smithi155.stdout: "chooseleaf_stable": 1, 2020-03-11T22:45:57.671 INFO:tasks.workunit.client.0.smithi155.stdout: "straw_calc_version": 1, 2020-03-11T22:45:57.671 INFO:tasks.workunit.client.0.smithi155.stdout: "allowed_bucket_algs": 54, 2020-03-11T22:45:57.672 INFO:tasks.workunit.client.0.smithi155.stdout: "profile": "jewel", 2020-03-11T22:45:57.672 INFO:tasks.workunit.client.0.smithi155.stdout: "optimal_tunables": 1, 2020-03-11T22:45:57.672 INFO:tasks.workunit.client.0.smithi155.stdout: "legacy_tunables": 0, 2020-03-11T22:45:57.672 INFO:tasks.workunit.client.0.smithi155.stdout: "minimum_required_version": "jewel", 2020-03-11T22:45:57.672 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables": 1, 2020-03-11T22:45:57.672 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables2": 1, 2020-03-11T22:45:57.672 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v2_rules": 1, 2020-03-11T22:45:57.673 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables3": 1, 2020-03-11T22:45:57.673 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v3_rules": 0, 2020-03-11T22:45:57.673 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v4_buckets": 1, 2020-03-11T22:45:57.673 INFO:tasks.workunit.client.0.smithi155.stdout: "require_feature_tunables5": 1, 2020-03-11T22:45:57.673 INFO:tasks.workunit.client.0.smithi155.stdout: "has_v5_rules": 0 2020-03-11T22:45:57.673 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.673 INFO:tasks.workunit.client.0.smithi155.stdout: "choose_args": {} 2020-03-11T22:45:57.674 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.674 INFO:tasks.workunit.client.0.smithi155.stdout: "fsmap": { 2020-03-11T22:45:57.674 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:45:57.674 INFO:tasks.workunit.client.0.smithi155.stdout: "default_fscid": -1, 2020-03-11T22:45:57.674 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": { 2020-03-11T22:45:57.674 INFO:tasks.workunit.client.0.smithi155.stdout: "compat": {}, 2020-03-11T22:45:57.674 INFO:tasks.workunit.client.0.smithi155.stdout: "ro_compat": {}, 2020-03-11T22:45:57.674 INFO:tasks.workunit.client.0.smithi155.stdout: "incompat": { 2020-03-11T22:45:57.675 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_1": "base v0.20", 2020-03-11T22:45:57.675 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_2": "client writeable ranges", 2020-03-11T22:45:57.675 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_3": "default file layouts on dirs", 2020-03-11T22:45:57.675 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_4": "dir inode in separate object", 2020-03-11T22:45:57.675 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_5": "mds uses versioned encoding", 2020-03-11T22:45:57.675 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_6": "dirfrag is stored in omap", 2020-03-11T22:45:57.675 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_8": "no anchor table", 2020-03-11T22:45:57.676 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_9": "file layout v2", 2020-03-11T22:45:57.676 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_10": "snaprealm v2" 2020-03-11T22:45:57.676 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.676 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.676 INFO:tasks.workunit.client.0.smithi155.stdout: "feature_flags": { 2020-03-11T22:45:57.676 INFO:tasks.workunit.client.0.smithi155.stdout: "enable_multiple": false, 2020-03-11T22:45:57.676 INFO:tasks.workunit.client.0.smithi155.stdout: "ever_enabled_multiple": false 2020-03-11T22:45:57.676 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.677 INFO:tasks.workunit.client.0.smithi155.stdout: "standbys": [], 2020-03-11T22:45:57.677 INFO:tasks.workunit.client.0.smithi155.stdout: "filesystems": [] 2020-03-11T22:45:57.677 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.677 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_first_committed": 1, 2020-03-11T22:45:57.677 INFO:tasks.workunit.client.0.smithi155.stdout: "mdsmap_last_committed": 1, 2020-03-11T22:45:57.677 INFO:tasks.workunit.client.0.smithi155.stdout: "auth": { 2020-03-11T22:45:57.677 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:45:57.678 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 5, 2020-03-11T22:45:57.678 INFO:tasks.workunit.client.0.smithi155.stdout: "num_secrets": 3 2020-03-11T22:45:57.678 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.678 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 1, 2020-03-11T22:45:57.678 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_active": 1, 2020-03-11T22:45:57.678 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_unknown": 0, 2020-03-11T22:45:57.678 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osd": 3, 2020-03-11T22:45:57.678 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_sum": { 2020-03-11T22:45:57.679 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:45:57.679 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 48430, 2020-03-11T22:45:57.679 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 101, 2020-03-11T22:45:57.679 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 1, 2020-03-11T22:45:57.679 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 303, 2020-03-11T22:45:57.679 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:45:57.679 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:45:57.679 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 101, 2020-03-11T22:45:57.680 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:45:57.680 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:45:57.680 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 101, 2020-03-11T22:45:57.680 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:45:57.680 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:45:57.680 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:45:57.680 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 101, 2020-03-11T22:45:57.681 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 101, 2020-03-11T22:45:57.681 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:45:57.681 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:45:57.681 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:45:57.681 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:45:57.681 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:45:57.681 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:45:57.682 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:45:57.682 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:45:57.682 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:45:57.682 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:45:57.682 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:45:57.682 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:45:57.682 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:45:57.683 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:45:57.683 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:45:57.683 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:45:57.683 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:45:57.683 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:45:57.683 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:45:57.683 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:45:57.683 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:45:57.684 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:45:57.684 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:45:57.684 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:45:57.684 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:45:57.684 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.684 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:45:57.684 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:45:57.685 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:45:57.685 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:45:57.685 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 0, 2020-03-11T22:45:57.685 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 0, 2020-03-11T22:45:57.685 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:45:57.685 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:57.685 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:45:57.686 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:45:57.686 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:45:57.686 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.686 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 18, 2020-03-11T22:45:57.686 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 18, 2020-03-11T22:45:57.686 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 3, 2020-03-11T22:45:57.686 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 3, 2020-03-11T22:45:57.686 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 0 2020-03-11T22:45:57.687 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.687 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum": { 2020-03-11T22:45:57.687 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:45:57.687 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:45:57.687 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 3, 2020-03-11T22:45:57.687 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:45:57.687 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:45:57.687 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:45:57.688 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:45:57.688 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3180608, 2020-03-11T22:45:57.688 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 34624, 2020-03-11T22:45:57.688 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 8, 2020-03-11T22:45:57.688 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145719, 2020-03-11T22:45:57.688 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311392192, 2020-03-11T22:45:57.688 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:45:57.689 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:45:57.689 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318865604608, 2020-03-11T22:45:57.689 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:45:57.689 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 35454976, 2020-03-11T22:45:57.689 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 1550952, 2020-03-11T22:45:57.689 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:45:57.689 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:57.690 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:45:57.690 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 8536, 2020-03-11T22:45:57.690 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221216936 2020-03-11T22:45:57.690 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.690 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:45:57.690 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:45:57.690 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:45:57.690 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2020-03-11T22:45:57.691 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:45:57.691 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:45:57.691 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:45:57.691 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.691 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:45:57.691 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:45:57.691 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:45:57.692 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:45:57.692 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:45:57.692 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.692 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:45:57.692 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:45:57.692 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.692 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_sum_by_class": { 2020-03-11T22:45:57.692 INFO:tasks.workunit.client.0.smithi155.stdout: "hdd": { 2020-03-11T22:45:57.693 INFO:tasks.workunit.client.0.smithi155.stdout: "up_from": 0, 2020-03-11T22:45:57.693 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 0, 2020-03-11T22:45:57.693 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pgs": 3, 2020-03-11T22:45:57.693 INFO:tasks.workunit.client.0.smithi155.stdout: "num_osds": 3, 2020-03-11T22:45:57.693 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_osds": 3, 2020-03-11T22:45:57.693 INFO:tasks.workunit.client.0.smithi155.stdout: "num_per_pool_omap_osds": 3, 2020-03-11T22:45:57.694 INFO:tasks.workunit.client.0.smithi155.stdout: "kb": 314572800, 2020-03-11T22:45:57.694 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used": 3180608, 2020-03-11T22:45:57.694 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_data": 34624, 2020-03-11T22:45:57.694 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_omap": 8, 2020-03-11T22:45:57.694 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_used_meta": 3145719, 2020-03-11T22:45:57.694 INFO:tasks.workunit.client.0.smithi155.stdout: "kb_avail": 311392192, 2020-03-11T22:45:57.694 INFO:tasks.workunit.client.0.smithi155.stdout: "statfs": { 2020-03-11T22:45:57.694 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 322122547200, 2020-03-11T22:45:57.695 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 318865604608, 2020-03-11T22:45:57.695 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 3221225472, 2020-03-11T22:45:57.695 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 35454976, 2020-03-11T22:45:57.695 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 1550952, 2020-03-11T22:45:57.695 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:45:57.695 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:57.695 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:45:57.695 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 8536, 2020-03-11T22:45:57.696 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 3221216936 2020-03-11T22:45:57.696 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.696 INFO:tasks.workunit.client.0.smithi155.stdout: "hb_peers": [], 2020-03-11T22:45:57.696 INFO:tasks.workunit.client.0.smithi155.stdout: "snap_trim_queue_len": 0, 2020-03-11T22:45:57.696 INFO:tasks.workunit.client.0.smithi155.stdout: "num_snap_trimming": 0, 2020-03-11T22:45:57.696 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shards_repaired": 0, 2020-03-11T22:45:57.696 INFO:tasks.workunit.client.0.smithi155.stdout: "op_queue_age_hist": { 2020-03-11T22:45:57.696 INFO:tasks.workunit.client.0.smithi155.stdout: "histogram": [], 2020-03-11T22:45:57.697 INFO:tasks.workunit.client.0.smithi155.stdout: "upper_bound": 1 2020-03-11T22:45:57.697 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.697 INFO:tasks.workunit.client.0.smithi155.stdout: "perf_stat": { 2020-03-11T22:45:57.697 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ms": 0, 2020-03-11T22:45:57.697 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ms": 0, 2020-03-11T22:45:57.697 INFO:tasks.workunit.client.0.smithi155.stdout: "commit_latency_ns": 0, 2020-03-11T22:45:57.697 INFO:tasks.workunit.client.0.smithi155.stdout: "apply_latency_ns": 0 2020-03-11T22:45:57.698 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.698 INFO:tasks.workunit.client.0.smithi155.stdout: "alerts": [], 2020-03-11T22:45:57.698 INFO:tasks.workunit.client.0.smithi155.stdout: "network_ping_times": [] 2020-03-11T22:45:57.698 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.698 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.698 INFO:tasks.workunit.client.0.smithi155.stdout: "pool_stats": [ 2020-03-11T22:45:57.698 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.699 INFO:tasks.workunit.client.0.smithi155.stdout: "poolid": 1, 2020-03-11T22:45:57.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg": 1, 2020-03-11T22:45:57.699 INFO:tasks.workunit.client.0.smithi155.stdout: "stat_sum": { 2020-03-11T22:45:57.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes": 48430, 2020-03-11T22:45:57.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects": 101, 2020-03-11T22:45:57.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_clones": 1, 2020-03-11T22:45:57.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_object_copies": 303, 2020-03-11T22:45:57.699 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing_on_primary": 0, 2020-03-11T22:45:57.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_missing": 0, 2020-03-11T22:45:57.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_degraded": 101, 2020-03-11T22:45:57.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_misplaced": 0, 2020-03-11T22:45:57.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_unfound": 0, 2020-03-11T22:45:57.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_dirty": 101, 2020-03-11T22:45:57.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_whiteouts": 0, 2020-03-11T22:45:57.700 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read": 0, 2020-03-11T22:45:57.701 INFO:tasks.workunit.client.0.smithi155.stdout: "num_read_kb": 0, 2020-03-11T22:45:57.701 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write": 101, 2020-03-11T22:45:57.701 INFO:tasks.workunit.client.0.smithi155.stdout: "num_write_kb": 101, 2020-03-11T22:45:57.701 INFO:tasks.workunit.client.0.smithi155.stdout: "num_scrub_errors": 0, 2020-03-11T22:45:57.701 INFO:tasks.workunit.client.0.smithi155.stdout: "num_shallow_scrub_errors": 0, 2020-03-11T22:45:57.701 INFO:tasks.workunit.client.0.smithi155.stdout: "num_deep_scrub_errors": 0, 2020-03-11T22:45:57.702 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_recovered": 0, 2020-03-11T22:45:57.702 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_recovered": 0, 2020-03-11T22:45:57.702 INFO:tasks.workunit.client.0.smithi155.stdout: "num_keys_recovered": 0, 2020-03-11T22:45:57.702 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_omap": 0, 2020-03-11T22:45:57.702 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_hit_set_archive": 0, 2020-03-11T22:45:57.702 INFO:tasks.workunit.client.0.smithi155.stdout: "num_bytes_hit_set_archive": 0, 2020-03-11T22:45:57.702 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush": 0, 2020-03-11T22:45:57.702 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_kb": 0, 2020-03-11T22:45:57.703 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict": 0, 2020-03-11T22:45:57.703 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_kb": 0, 2020-03-11T22:45:57.703 INFO:tasks.workunit.client.0.smithi155.stdout: "num_promote": 0, 2020-03-11T22:45:57.703 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_high": 0, 2020-03-11T22:45:57.703 INFO:tasks.workunit.client.0.smithi155.stdout: "num_flush_mode_low": 0, 2020-03-11T22:45:57.703 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_some": 0, 2020-03-11T22:45:57.703 INFO:tasks.workunit.client.0.smithi155.stdout: "num_evict_mode_full": 0, 2020-03-11T22:45:57.704 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_pinned": 0, 2020-03-11T22:45:57.704 INFO:tasks.workunit.client.0.smithi155.stdout: "num_legacy_snapsets": 0, 2020-03-11T22:45:57.704 INFO:tasks.workunit.client.0.smithi155.stdout: "num_large_omap_objects": 0, 2020-03-11T22:45:57.704 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_manifest": 0, 2020-03-11T22:45:57.704 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_bytes": 0, 2020-03-11T22:45:57.704 INFO:tasks.workunit.client.0.smithi155.stdout: "num_omap_keys": 0, 2020-03-11T22:45:57.705 INFO:tasks.workunit.client.0.smithi155.stdout: "num_objects_repaired": 0 2020-03-11T22:45:57.705 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.705 INFO:tasks.workunit.client.0.smithi155.stdout: "store_stats": { 2020-03-11T22:45:57.705 INFO:tasks.workunit.client.0.smithi155.stdout: "total": 0, 2020-03-11T22:45:57.705 INFO:tasks.workunit.client.0.smithi155.stdout: "available": 0, 2020-03-11T22:45:57.705 INFO:tasks.workunit.client.0.smithi155.stdout: "internally_reserved": 0, 2020-03-11T22:45:57.706 INFO:tasks.workunit.client.0.smithi155.stdout: "allocated": 19857408, 2020-03-11T22:45:57.706 INFO:tasks.workunit.client.0.smithi155.stdout: "data_stored": 1241088, 2020-03-11T22:45:57.706 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed": 0, 2020-03-11T22:45:57.706 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_allocated": 0, 2020-03-11T22:45:57.706 INFO:tasks.workunit.client.0.smithi155.stdout: "data_compressed_original": 0, 2020-03-11T22:45:57.706 INFO:tasks.workunit.client.0.smithi155.stdout: "omap_allocated": 0, 2020-03-11T22:45:57.706 INFO:tasks.workunit.client.0.smithi155.stdout: "internal_metadata": 0 2020-03-11T22:45:57.707 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.707 INFO:tasks.workunit.client.0.smithi155.stdout: "log_size": 18, 2020-03-11T22:45:57.707 INFO:tasks.workunit.client.0.smithi155.stdout: "ondisk_log_size": 18, 2020-03-11T22:45:57.707 INFO:tasks.workunit.client.0.smithi155.stdout: "up": 3, 2020-03-11T22:45:57.707 INFO:tasks.workunit.client.0.smithi155.stdout: "acting": 3, 2020-03-11T22:45:57.707 INFO:tasks.workunit.client.0.smithi155.stdout: "num_store_stats": 3 2020-03-11T22:45:57.707 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.707 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.708 INFO:tasks.workunit.client.0.smithi155.stdout: "osd_stats": [ 2020-03-11T22:45:57.708 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.708 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:45:57.708 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 171798691850 2020-03-11T22:45:57.708 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.708 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.709 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:45:57.709 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 38654705683 2020-03-11T22:45:57.709 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.709 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.709 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:45:57.709 INFO:tasks.workunit.client.0.smithi155.stdout: "seq": 184683593737 2020-03-11T22:45:57.709 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.709 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.710 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_state": [ 2020-03-11T22:45:57.710 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.710 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "active+undersized+degraded", 2020-03-11T22:45:57.710 INFO:tasks.workunit.client.0.smithi155.stdout: "num": 1 2020-03-11T22:45:57.710 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.710 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.710 INFO:tasks.workunit.client.0.smithi155.stdout: "num_pg_by_osd": [ 2020-03-11T22:45:57.710 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.711 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2, 2020-03-11T22:45:57.711 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:45:57.711 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 1, 2020-03-11T22:45:57.711 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:57.711 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.711 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.711 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 2147483647, 2020-03-11T22:45:57.711 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:45:57.712 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 1, 2020-03-11T22:45:57.712 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:57.712 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.712 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.712 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 1, 2020-03-11T22:45:57.712 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 0, 2020-03-11T22:45:57.712 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 0, 2020-03-11T22:45:57.713 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:57.713 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.713 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.713 INFO:tasks.workunit.client.0.smithi155.stdout: "osd": 0, 2020-03-11T22:45:57.713 INFO:tasks.workunit.client.0.smithi155.stdout: "num_primary_pg": 1, 2020-03-11T22:45:57.713 INFO:tasks.workunit.client.0.smithi155.stdout: "num_acting_pg": 1, 2020-03-11T22:45:57.713 INFO:tasks.workunit.client.0.smithi155.stdout: "num_up_not_acting_pg": 0 2020-03-11T22:45:57.713 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.714 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.714 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": [ 2020-03-11T22:45:57.714 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:45:57.714 INFO:tasks.workunit.client.0.smithi155.stdout: "pool": 1, 2020-03-11T22:45:57.714 INFO:tasks.workunit.client.0.smithi155.stdout: "purged_snaps": {} 2020-03-11T22:45:57.714 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.714 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:45:57.715 INFO:tasks.workunit.client.0.smithi155.stdout: "servicemap": { 2020-03-11T22:45:57.715 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": 1, 2020-03-11T22:45:57.715 INFO:tasks.workunit.client.0.smithi155.stdout: "modified": "2020-03-11T22:43:30.109994+0000", 2020-03-11T22:45:57.715 INFO:tasks.workunit.client.0.smithi155.stdout: "services": {} 2020-03-11T22:45:57.715 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:45:57.715 INFO:tasks.workunit.client.0.smithi155.stdout: "paxos": { 2020-03-11T22:45:57.715 INFO:tasks.workunit.client.0.smithi155.stdout: "first_committed": 1, 2020-03-11T22:45:57.716 INFO:tasks.workunit.client.0.smithi155.stdout: "last_committed": 423, 2020-03-11T22:45:57.716 INFO:tasks.workunit.client.0.smithi155.stdout: "last_pn": 0, 2020-03-11T22:45:57.716 INFO:tasks.workunit.client.0.smithi155.stdout: "accepted_pn": 0 2020-03-11T22:45:57.716 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:45:57.716 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:45:57.717 INFO:tasks.workunit.client.0.smithi155.stderr:report 3371223272 2020-03-11T22:45:57.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1603: wait_for_clean: return 1 2020-03-11T22:45:57.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:341: TEST_divergent_ec: echo 'delay recovery' 2020-03-11T22:45:57.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:342: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:57.718 INFO:tasks.workunit.client.0.smithi155.stdout:delay recovery 2020-03-11T22:45:57.883 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:57.884 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:45:22.967124+0000 38'102 44:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:45:57.884 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:45:57.884 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:57.885 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:45:57.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:45:57.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.0 2020-03-11T22:45:57.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:45:57.894 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:45:57.894 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:57.894 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:57.895 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:45:57.895 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.0.asok 2020-03-11T22:45:57.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:45:57.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.0.asok set_recovery_delay 100000 2020-03-11T22:45:57.986 INFO:tasks.workunit.client.0.smithi155.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:45:57.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:343: TEST_divergent_ec: for i in $non_divergent 2020-03-11T22:45:57.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: get_asok_path osd.2 2020-03-11T22:45:57.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.2 2020-03-11T22:45:57.994 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.2 ']' 2020-03-11T22:45:57.995 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:45:57.995 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:57.995 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:45:57.995 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.2.asok 2020-03-11T22:45:57.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:45:57.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:345: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.2.asok set_recovery_delay 100000 2020-03-11T22:45:58.086 INFO:tasks.workunit.client.0.smithi155.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:45:58.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:349: TEST_divergent_ec: echo 'revive divergent 1' 2020-03-11T22:45:58.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:350: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:45:58.093 INFO:tasks.workunit.client.0.smithi155.stdout:revive divergent 1 2020-03-11T22:45:58.412 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:45:58.413 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:45:22.967124+0000 38'102 44:137 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:45:58.413 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:45:58.413 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:45:58.414 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:45:58.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:351: TEST_divergent_ec: ceph osd set noup 2020-03-11T22:45:59.073 INFO:tasks.workunit.client.0.smithi155.stderr:noup is set 2020-03-11T22:45:59.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:352: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:45:59.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:45:59.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:45:59.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:45:59.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:45:59.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:45:59.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:45:59.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:45:59.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:45:59.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:45:59.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:45:59.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:45:59.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:45:59.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:45:59.091 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:45:59.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:45:59.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:45:59.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:45:59.092 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:45:59.092 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:45:59.092 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:45:59.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:45:59.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:45:59.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:45:59.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:45:59.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:45:59.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:45:59.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:45:59.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:59.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:45:59.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:45:59.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:45:59.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:45:59.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:45:59.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:45:59.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:45:59.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:45:59.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:45:59.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:45:59.095 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:45:59.111 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:59.108+0000 7fac86051ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:59.112 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:59.110+0000 7fac86051ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:59.113 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:59.111+0000 7fac86051ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:45:59.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:876: activate_osd: return 0 2020-03-11T22:45:59.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:353: TEST_divergent_ec: sleep 5 2020-03-11T22:45:59.629 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:45:59.627+0000 7fac86051ec0 -1 Falling back to public interface 2020-03-11T22:46:00.514 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:00.511+0000 7fac86051ec0 -1 osd.1 28 log_to_monitors {default=true} 2020-03-11T22:46:04.435 INFO:tasks.workunit.client.0.smithi155.stdout:delay recovery divergent 2020-03-11T22:46:04.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:355: TEST_divergent_ec: echo 'delay recovery divergent' 2020-03-11T22:46:04.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:356: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:04.767 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:04.768 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 0 101 0 0 48430 0 0 18 18 active+undersized+degraded 2020-03-11T22:45:22.967124+0000 38'102 44:138 [NONE,0,2] 0 [NONE,0,2] 0 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:46:04.768 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:46:04.768 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:04.769 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:46:04.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: get_asok_path osd.1 2020-03-11T22:46:04.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:46:04.778 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:46:04.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:04.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:04.779 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:46:04.779 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.17464/ceph-osd.1.asok 2020-03-11T22:46:04.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: CEPH_ARGS= 2020-03-11T22:46:04.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:357: TEST_divergent_ec: ceph --admin-daemon /tmp/ceph-asok.17464/ceph-osd.1.asok set_recovery_delay 100000 2020-03-11T22:46:04.870 INFO:tasks.workunit.client.0.smithi155.stdout:set_recovery_delay: set osd_recovery_delay_start to 100000 2020-03-11T22:46:04.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:359: TEST_divergent_ec: ceph osd unset noup 2020-03-11T22:46:05.633 INFO:tasks.workunit.client.0.smithi155.stderr:noup is unset 2020-03-11T22:46:05.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:361: TEST_divergent_ec: wait_for_osd up 0 2020-03-11T22:46:05.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:05.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:46:05.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:05.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:05.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:05.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:05.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:05.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:05.643 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:46:05.999 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 40 up_thru 43 down_at 39 last_clean_interval [34,38) [v2:127.0.0.1:6802/50341,v1:127.0.0.1:6803/50341] [v2:127.0.0.1:6804/50341,v1:127.0.0.1:6805/50341] exists,up e0e5724e-7440-465c-9db6-82e8c737bcc8 2020-03-11T22:46:05.999 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:46:05.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:05.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:362: TEST_divergent_ec: wait_for_osd up 1 2020-03-11T22:46:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:46:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:06.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:06.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:06.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:06.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:06.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:06.344 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6818/51969,v1:127.0.0.1:6819/51969] [v2:127.0.0.1:6820/51969,v1:127.0.0.1:6821/51969] exists,up 7b6947d8-603c-41a4-abcb-8b50325a2007 2020-03-11T22:46:06.344 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:46:06.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:06.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:06.344 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:06.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:363: TEST_divergent_ec: wait_for_osd up 2 2020-03-11T22:46:06.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:06.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:46:06.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:06.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:06.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:06.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:06.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:06.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:46:06.686 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 43 up_thru 0 down_at 42 last_clean_interval [35,38) [v2:127.0.0.1:6810/50581,v1:127.0.0.1:6811/50581] [v2:127.0.0.1:6812/50581,v1:127.0.0.1:6813/50581] exists,up 797c47f4-417e-4663-8ca9-e4cc78330dd9 2020-03-11T22:46:06.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:06.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:06.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:06.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:365: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:07.017 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:07.018 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 peering 2020-03-11T22:46:05.506191+0000 28'104 48:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:46:07.018 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:46:07.018 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:07.021 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:46:07.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:366: TEST_divergent_ec: echo 'wait for peering' 2020-03-11T22:46:07.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:367: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:07.027 INFO:tasks.workunit.client.0.smithi155.stdout:wait for peering 2020-03-11T22:46:07.347 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:07.347 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 0 0 0 0 48783 0 0 14 14 peering 2020-03-11T22:46:05.506191+0000 28'104 48:119 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:46:07.347 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:46:07.348 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:07.348 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:46:07.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:368: TEST_divergent_ec: rados -p test put foo /etc/fstab 2020-03-11T22:46:19.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:370: TEST_divergent_ec: echo 'killing divergent 1' 2020-03-11T22:46:19.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:371: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:19.047 INFO:tasks.workunit.client.0.smithi155.stdout:killing divergent 1 2020-03-11T22:46:19.379 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:19.380 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:46:05.915222+0000 38'102 50:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:46:19.380 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:46:19.380 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:19.381 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:46:19.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:372: TEST_divergent_ec: kill_daemons td/divergent-priors KILL osd.1 2020-03-11T22:46:19.390 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:19.390 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:19.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:19.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:19.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:19.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:19.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:374: TEST_divergent_ec: echo 'reviving divergent 1' 2020-03-11T22:46:19.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:375: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:19.496 INFO:tasks.workunit.client.0.smithi155.stdout:reviving divergent 1 2020-03-11T22:46:19.816 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:19.816 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 101 4 4 0 0 48430 0 0 12 12 active+recovering+degraded+wait 2020-03-11T22:46:05.915222+0000 38'102 50:130 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:46:19.816 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:46:19.817 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:19.818 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:46:19.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:376: TEST_divergent_ec: activate_osd td/divergent-priors 1 2020-03-11T22:46:19.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/divergent-priors 2020-03-11T22:46:19.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:46:19.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:46:19.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:46:19.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/divergent-priors/1 2020-03-11T22:46:19.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 ' 2020-03-11T22:46:19.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:19.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:19.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:19.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/divergent-priors/1' 2020-03-11T22:46:19.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/divergent-priors/1/journal' 2020-03-11T22:46:19.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:19.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:46:19.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/divergent-priors' 2020-03-11T22:46:19.831 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:46:19.831 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:19.831 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:19.832 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:19.832 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:19.832 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:46:19.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:46:19.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' 2020-03-11T22:46:19.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:19.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/divergent-priors/$name.log' 2020-03-11T22:46:19.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/divergent-priors/$name.pid' 2020-03-11T22:46:19.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:19.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:19.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:19.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:46:19.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:46:19.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/divergent-priors/1 2020-03-11T22:46:19.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:46:19.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=02364139-af2c-42f4-b33a-c83055e85709 --auth-supported=none --mon-host=127.0.0.1:7115 --osd_force_auth_primary_missing_objects=1000000 --osd_debug_pg_log_writeout=true --osd_min_pg_log_entries=12 --osd_max_pg_log_entries=12 --osd_pg_log_trim_min=6 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/divergent-priors/1 --osd-journal=td/divergent-priors/1/journal --chdir= --run-dir=td/divergent-priors '--admin-socket=/tmp/ceph-asok.17464/$cluster-$name.asok' --debug-osd=20 '--log-file=td/divergent-priors/$name.log' '--pid-file=td/divergent-priors/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:19.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/divergent-priors/1/whoami 2020-03-11T22:46:19.834 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:46:19.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:46:19.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:46:19.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:46:19.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:46:19.855 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:19.852+0000 7f4015cceec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:19.857 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:19.853+0000 7f4015cceec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:19.858 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:19.855+0000 7f4015cceec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:20.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:46:20.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:20.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:46:20.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:20.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:20.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:20.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:20.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:20.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:20.176 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:46:20.374 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:20.370+0000 7f4015cceec0 -1 Falling back to public interface 2020-03-11T22:46:20.555 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 48 up_thru 48 down_at 30 last_clean_interval [9,28) [v2:127.0.0.1:6818/51969,v1:127.0.0.1:6819/51969] [v2:127.0.0.1:6820/51969,v1:127.0.0.1:6821/51969] exists,up 7b6947d8-603c-41a4-abcb-8b50325a2007 2020-03-11T22:46:20.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:20.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:20.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:20.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:378: TEST_divergent_ec: sleep 20 2020-03-11T22:46:21.249 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:21.245+0000 7f4015cceec0 -1 osd.1 50 log_to_monitors {default=true} 2020-03-11T22:46:40.556 INFO:tasks.workunit.client.0.smithi155.stdout:allowing recovery 2020-03-11T22:46:40.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:380: TEST_divergent_ec: echo 'allowing recovery' 2020-03-11T22:46:40.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:381: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:40.883 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:40.883 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:46:22.542578+0000 50'103 53:154 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:46:40.883 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:46:40.884 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:40.885 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:46:40.893 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:40.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.0 debug kick_recovery_wq 0 2020-03-11T22:46:41.015 INFO:tasks.workunit.client.0.smithi155.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:46:41.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:41.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.1 debug kick_recovery_wq 0 2020-03-11T22:46:41.144 INFO:tasks.workunit.client.0.smithi155.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:46:41.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:383: TEST_divergent_ec: for i in $osds 2020-03-11T22:46:41.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:385: TEST_divergent_ec: ceph tell osd.2 debug kick_recovery_wq 0 2020-03-11T22:46:41.274 INFO:tasks.workunit.client.0.smithi155.stderr:kicking recovery queue. set osd_recovery_delay_start to 0 2020-03-11T22:46:41.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:388: TEST_divergent_ec: echo 'reading divergent objects' 2020-03-11T22:46:41.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:389: TEST_divergent_ec: ceph pg dump pgs 2020-03-11T22:46:41.282 INFO:tasks.workunit.client.0.smithi155.stdout:reading divergent objects 2020-03-11T22:46:41.602 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:46:41.602 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 102 0 0 0 0 48913 0 0 13 13 active+clean 2020-03-11T22:46:22.542578+0000 50'103 53:154 [1,0,2] 1 [1,0,2] 1 0'0 2020-03-11T22:43:52.885379+0000 0'0 2020-03-11T22:43:52.885379+0000 0 2020-03-11T22:46:41.602 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:46:41.603 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:46:41.604 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:46:41.612 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: expr 2 + 2 2020-03-11T22:46:41.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: seq 1 4 2020-03-11T22:46:41.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:46:41.614 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_1 td/divergent-priors/existing 2020-03-11T22:46:41.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:46:41.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_2 td/divergent-priors/existing 2020-03-11T22:46:41.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:46:41.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_3 td/divergent-priors/existing 2020-03-11T22:46:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:390: TEST_divergent_ec: for i in $(seq 1 $(expr $DIVERGENT_WRITE + $DIVERGENT_REMOVE)) 2020-03-11T22:46:41.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:392: TEST_divergent_ec: rados -p test get existing_4 td/divergent-priors/existing 2020-03-11T22:46:41.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:394: TEST_divergent_ec: rm -f td/divergent-priors/existing 2020-03-11T22:46:41.753 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:46:41.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:396: TEST_divergent_ec: grep _merge_object_divergent_entries td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:46:41.758 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:1 entries: 28'103 (27'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:44:11.921913+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:46:41.758 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 20 _merge_object_divergent_entries: keeping 28'103 (27'99) clone 1:7a80c0d5:::existing_99:1 by unknown.0.0:0 2020-03-11T22:44:11.921913+0000 0 snaps [1] ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 2020-03-11T22:46:41.758 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:46:41.758 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version: 27'99 first_divergent_update: 28'103 last_divergent_update: 28'103 2020-03-11T22:46:41.758 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 has no more recent entries in log 2020-03-11T22:46:41.758 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:1 prior_version or op type indicates creation, deleting 2020-03-11T22:46:41.759 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 20 _merge_object_divergent_entries: merging hoid 1:7a80c0d5:::existing_99:head entries: 28'104 (27'99) modify 1:7a80c0d5:::existing_99:head by client.4553.0:1 2020-03-11T22:44:18.676601+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:41.759 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 20 _merge_object_divergent_entries: keeping 28'104 (27'99) modify 1:7a80c0d5:::existing_99:head by client.4553.0:1 2020-03-11T22:44:18.676601+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:41.759 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:46:41.759 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head prior_version: 27'99 first_divergent_update: 28'104 last_divergent_update: 28'104 2020-03-11T22:46:41.759 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head has no more recent entries in log 2020-03-11T22:46:41.759 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:46:41.759 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head olog_can_rollback_to: 28'104 2020-03-11T22:46:41.760 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot rollback 28'104 (27'99) modify 1:7a80c0d5:::existing_99:head by client.4553.0:1 2020-03-11T22:44:18.676601+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:41.760 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:7a80c0d5:::existing_99:head cannot roll back, removing and adding to missing 2020-03-11T22:46:41.760 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 20 _merge_object_divergent_entries: merging hoid 1:c156033e:::existing_100:head entries: 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4544.0:1 2020-03-11T22:44:17.502633+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0,27'102 (27'101) modify 1:c156033e:::existing_100:head by client.4550.0:1 2020-03-11T22:44:18.509260+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:41.760 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 20 _merge_object_divergent_entries: keeping 27'101 (27'100) modify 1:c156033e:::existing_100:head by client.4544.0:1 2020-03-11T22:44:17.502633+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:41.761 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 20 _merge_object_divergent_entries: keeping 27'102 (27'101) modify 1:c156033e:::existing_100:head by client.4550.0:1 2020-03-11T22:44:18.509260+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:41.761 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid object_not_in_store: 0 2020-03-11T22:46:41.761 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head prior_version: 27'100 first_divergent_update: 27'101 last_divergent_update: 27'102 2020-03-11T22:46:41.761 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head has no more recent entries in log 2020-03-11T22:46:41.761 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head must be rolled back or recovered, attempting to rollback 2020-03-11T22:46:41.761 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head olog_can_rollback_to: 28'104 2020-03-11T22:46:41.762 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot rollback 27'102 (27'101) modify 1:c156033e:::existing_100:head by client.4550.0:1 2020-03-11T22:44:18.509260+0000 0 ObjectCleanRegions clean_offsets: [483~18446744073709551132], clean_omap: 1, new_object: 0 2020-03-11T22:46:41.762 INFO:tasks.workunit.client.0.smithi155.stdout:td/divergent-priors/osd.1.log:2020-03-11T22:46:05.857+0000 7fac6420c700 10 _merge_object_divergent_entries: hoid 1:c156033e:::existing_100:head cannot roll back, removing and adding to missing 2020-03-11T22:46:41.762 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: find td/divergent-priors -name '*osd*log' 2020-03-11T22:46:41.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:405: TEST_divergent_ec: grep -q '_merge_object_divergent_entries.*rolled back' td/divergent-priors/osd.0.log td/divergent-priors/osd.1.log td/divergent-priors/osd.2.log 2020-03-11T22:46:41.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:410: TEST_divergent_ec: echo success 2020-03-11T22:46:41.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:412: TEST_divergent_ec: delete_pool test 2020-03-11T22:46:41.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test 2020-03-11T22:46:41.766 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test test --yes-i-really-really-mean-it 2020-03-11T22:46:41.766 INFO:tasks.workunit.client.0.smithi155.stdout:success 2020-03-11T22:46:42.235 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test' does not exist 2020-03-11T22:46:42.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:413: TEST_divergent_ec: kill_daemons td/divergent-priors 2020-03-11T22:46:42.243 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:42.243 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:42.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:42.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:42.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:42.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:42.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/divergent-priors.sh:47: run: teardown td/divergent-priors 2020-03-11T22:46:42.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:46:42.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:42.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:46:42.352 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:42.352 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:42.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:42.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:42.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:42.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:42.356 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:42.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:42.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:42.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:42.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:42.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:42.362 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:42.362 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:42.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:42.363 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:42.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:42.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:42.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:46:42.375 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:42.375 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:42.375 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:46:42.375 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17464 2020-03-11T22:46:42.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:42.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:42.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:46:42.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/divergent-priors 0 2020-03-11T22:46:42.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/divergent-priors 2020-03-11T22:46:42.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:46:42.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/divergent-priors KILL 2020-03-11T22:46:42.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:42.378 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:42.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:42.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:42.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:42.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:42.379 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:42.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:42.380 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:42.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:42.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:42.381 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:42.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:42.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:42.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:42.383 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:42.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:42.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:46:42.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/divergent-priors 2020-03-11T22:46:42.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:42.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:42.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.17464 2020-03-11T22:46:42.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.17464 2020-03-11T22:46:42.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:42.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:42.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:46:42.388 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:46:42.388 INFO:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:46:42.456 INFO:tasks.workunit:Running workunit osd/ec-error-rollforward.sh... 2020-03-11T22:46:42.456 INFO:teuthology.orchestra.run.smithi155:workunit test osd/ec-error-rollforward.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh 2020-03-11T22:46:42.485 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:46:42.493 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:46:42.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:46:42.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:46:42.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:46:42.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:46:42.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:46:42.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:46:42.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:46:42.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:46:42.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/ec-error-rollforward 2020-03-11T22:46:42.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:6: run: local dir=td/ec-error-rollforward 2020-03-11T22:46:42.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:7: run: shift 2020-03-11T22:46:42.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: export CEPH_MON=127.0.0.1:7132 2020-03-11T22:46:42.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:10: run: CEPH_MON=127.0.0.1:7132 2020-03-11T22:46:42.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:11: run: export CEPH_ARGS 2020-03-11T22:46:42.495 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: uuidgen 2020-03-11T22:46:42.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:12: run: CEPH_ARGS+='--fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none ' 2020-03-11T22:46:42.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:13: run: CEPH_ARGS+='--mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:42.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: export margin=10 2020-03-11T22:46:42.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:14: run: margin=10 2020-03-11T22:46:42.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: export objects=200 2020-03-11T22:46:42.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:15: run: objects=200 2020-03-11T22:46:42.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: export poolname=test 2020-03-11T22:46:42.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:16: run: poolname=test 2020-03-11T22:46:42.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: set 2020-03-11T22:46:42.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:46:42.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:18: run: local funcs=TEST_ec_error_rollforward 2020-03-11T22:46:42.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:19: run: for func in $funcs 2020-03-11T22:46:42.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:20: run: setup td/ec-error-rollforward 2020-03-11T22:46:42.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/ec-error-rollforward 2020-03-11T22:46:42.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/ec-error-rollforward 2020-03-11T22:46:42.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:46:42.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:46:42.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:46:42.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:46:42.501 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:46:42.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:46:42.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:46:42.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:46:42.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:46:42.502 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:46:42.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:46:42.502 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:46:42.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:46:42.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:46:42.503 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:46:42.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:42.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:46:42.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:46:42.503 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:46:42.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:46:42.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:46:42.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:46:42.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:46:42.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:42.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:46:42.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52897 2020-03-11T22:46:42.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:46:42.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:46:42.506 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/ec-error-rollforward 2020-03-11T22:46:42.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:46:42.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:42.506 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:46:42.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.52897 2020-03-11T22:46:42.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:46:42.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:46:42.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:46:42.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:46:42.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/ec-error-rollforward 1' TERM HUP INT 2020-03-11T22:46:42.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:21: run: TEST_ec_error_rollforward td/ec-error-rollforward 2020-03-11T22:46:42.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:27: TEST_ec_error_rollforward: local dir=td/ec-error-rollforward 2020-03-11T22:46:42.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:28: TEST_ec_error_rollforward: run_mon td/ec-error-rollforward a 2020-03-11T22:46:42.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/ec-error-rollforward 2020-03-11T22:46:42.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:46:42.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:46:42.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:46:42.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/ec-error-rollforward/a 2020-03-11T22:46:42.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/ec-error-rollforward/a --run-dir=td/ec-error-rollforward 2020-03-11T22:46:42.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:46:42.816 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:42.816 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:42.816 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:42.816 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:42.816 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:46:42.816 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:46:42.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ec-error-rollforward/a '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --mon-cluster-log-file=td/ec-error-rollforward/log --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:46:43.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:46:43.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:46:43.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:43.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:43.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:46:43.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:46:43.003 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:43.003 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:43.003 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:43.004 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:43.004 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:43.004 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:46:43.004 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52897/ceph-mon.a.asok 2020-03-11T22:46:43.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:43.004 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52897/ceph-mon.a.asok config get fsid 2020-03-11T22:46:43.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:46:43.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:46:43.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:46:43.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:46:43.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:46:43.090 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:46:43.090 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:46:43.090 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:46:43.090 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:46:43.091 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:43.091 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:46:43.091 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.52897/ceph-mon.a.asok 2020-03-11T22:46:43.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:46:43.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.52897/ceph-mon.a.asok config get mon_host 2020-03-11T22:46:43.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:29: TEST_ec_error_rollforward: run_mgr td/ec-error-rollforward x 2020-03-11T22:46:43.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/ec-error-rollforward 2020-03-11T22:46:43.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:46:43.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:46:43.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:46:43.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/ec-error-rollforward/x 2020-03-11T22:46:43.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:46:44.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:46:44.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:44.050 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:44.050 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:44.050 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:44.051 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:46:44.051 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:46:44.051 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:44.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ec-error-rollforward/x '--log-file=td/ec-error-rollforward/$name.log' '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --run-dir=td/ec-error-rollforward '--pid-file=td/ec-error-rollforward/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:46:44.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:30: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 0 2020-03-11T22:46:44.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:44.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:44.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:46:44.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:44.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:46:44.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:44.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:44.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:44.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:44.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:46:44.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:46:44.074 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:44.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:44.075 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:44.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:44.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:44.075 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:44.075 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:44.075 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:44.076 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:46:44.076 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:46:44.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:46:44.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:44.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:44.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:44.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:44.076 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:44.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:44.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:44.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:44.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:44.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:44.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:46:44.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:44.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=91d53f57-7a3f-4003-a1b6-9fcff03b9c24 2020-03-11T22:46:44.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 91d53f57-7a3f-4003-a1b6-9fcff03b9c24' 2020-03-11T22:46:44.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:44.079 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 91d53f57-7a3f-4003-a1b6-9fcff03b9c24 2020-03-11T22:46:44.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBUamle4r/2BBAAHu6Lk+a3TH0rgrJWDYLFww== 2020-03-11T22:46:44.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBUamle4r/2BBAAHu6Lk+a3TH0rgrJWDYLFww=="}' 2020-03-11T22:46:44.086 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 91d53f57-7a3f-4003-a1b6-9fcff03b9c24 -i td/ec-error-rollforward/0/new.json 2020-03-11T22:46:44.464 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:46:44.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/0/new.json 2020-03-11T22:46:44.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBUamle4r/2BBAAHu6Lk+a3TH0rgrJWDYLFww== --osd-uuid 91d53f57-7a3f-4003-a1b6-9fcff03b9c24 2020-03-11T22:46:44.494 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:44.492+0000 7f16703a0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:44.496 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:44.493+0000 7f16703a0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:44.497 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:44.495+0000 7f16703a0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:44.498 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:44.495+0000 7f16703a0ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:46:44.498 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:44.495+0000 7f16703a0ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:46:44.498 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:44.495+0000 7f16703a0ec0 -1 bluestore(td/ec-error-rollforward/0/block) _read_bdev_label failed to open td/ec-error-rollforward/0/block: (2) No such file or directory 2020-03-11T22:46:44.541 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:44.538+0000 7f16703a0ec0 -1 bluestore(td/ec-error-rollforward/0) _read_fsid unparsable uuid 2020-03-11T22:46:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/0/keyring 2020-03-11T22:46:47.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:47.583 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2020-03-11T22:46:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:46:47.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:48.051 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:46:48.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:46:48.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:48.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:48.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:48.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:48.070 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:48.067+0000 7fba365e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:48.071 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:48.069+0000 7fba365e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:48.073 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:48.070+0000 7fba365e0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:48.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:46:48.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:48.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:46:48.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:48.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:48.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:48.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:48.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:48.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:48.403 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:46:48.588 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:48.586+0000 7fba365e0ec0 -1 Falling back to public interface 2020-03-11T22:46:48.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:49.418 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:49.415+0000 7fba365e0ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:46:49.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:49.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:49.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:49.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:49.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:46:49.749 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:46:50.093 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53326,v1:127.0.0.1:6803/53326] [v2:127.0.0.1:6804/53326,v1:127.0.0.1:6805/53326] exists,up 91d53f57-7a3f-4003-a1b6-9fcff03b9c24 2020-03-11T22:46:50.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:50.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:50.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:50.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:31: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 1 2020-03-11T22:46:50.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:50.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:50.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:46:50.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:50.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:46:50.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:50.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:50.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:50.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:50.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:46:50.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:46:50.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:50.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:50.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:50.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:50.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:50.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:50.097 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:50.097 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:50.097 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:46:50.097 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:46:50.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:46:50.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:50.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:50.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:50.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:50.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:50.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:50.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:50.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:50.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:50.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:50.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:46:50.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:50.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=78203e12-07b3-4e1f-bc02-ed97fd0fcbb8 2020-03-11T22:46:50.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 78203e12-07b3-4e1f-bc02-ed97fd0fcbb8' 2020-03-11T22:46:50.100 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:50.100 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 78203e12-07b3-4e1f-bc02-ed97fd0fcbb8 2020-03-11T22:46:50.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBaamlevBErBhAAO0bTj6wyqA1PPIXxBjgFKA== 2020-03-11T22:46:50.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBaamlevBErBhAAO0bTj6wyqA1PPIXxBjgFKA=="}' 2020-03-11T22:46:50.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 78203e12-07b3-4e1f-bc02-ed97fd0fcbb8 -i td/ec-error-rollforward/1/new.json 2020-03-11T22:46:50.580 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:46:50.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/1/new.json 2020-03-11T22:46:50.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBaamlevBErBhAAO0bTj6wyqA1PPIXxBjgFKA== --osd-uuid 78203e12-07b3-4e1f-bc02-ed97fd0fcbb8 2020-03-11T22:46:50.610 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:50.608+0000 7f2dee21bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:50.612 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:50.609+0000 7f2dee21bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:50.614 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:50.611+0000 7f2dee21bec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:50.615 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:50.612+0000 7f2dee21bec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:46:50.615 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:50.612+0000 7f2dee21bec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:46:50.615 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:50.612+0000 7f2dee21bec0 -1 bluestore(td/ec-error-rollforward/1/block) _read_bdev_label failed to open td/ec-error-rollforward/1/block: (2) No such file or directory 2020-03-11T22:46:50.687 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:50.685+0000 7f2dee21bec0 -1 bluestore(td/ec-error-rollforward/1) _read_fsid unparsable uuid 2020-03-11T22:46:53.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/1/keyring 2020-03-11T22:46:53.705 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:46:53.706 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2020-03-11T22:46:53.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:46:53.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:46:54.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:46:54.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:54.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:46:54.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:46:54.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:46:54.164 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:46:54.182 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:54.179+0000 7f5a0cc56ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:54.183 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:54.181+0000 7f5a0cc56ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:54.184 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:54.182+0000 7f5a0cc56ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:54.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:46:54.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:46:54.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:46:54.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:46:54.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:46:54.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:54.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:46:54.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:54.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:54.510 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:46:54.704 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:54.701+0000 7f5a0cc56ec0 -1 Falling back to public interface 2020-03-11T22:46:54.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:55.453 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:55.451+0000 7f5a0cc56ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:46:55.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:55.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:55.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:46:55.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:55.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:55.849 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:46:56.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:46:57.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:46:57.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:46:57.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:46:57.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:46:57.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:46:57.209 INFO:tasks.workunit.client.0.smithi155.stdout:2 2020-03-11T22:46:57.551 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53681,v1:127.0.0.1:6811/53681] [v2:127.0.0.1:6812/53681,v1:127.0.0.1:6813/53681] exists,up 78203e12-07b3-4e1f-bc02-ed97fd0fcbb8 2020-03-11T22:46:57.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:46:57.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:46:57.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:46:57.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:32: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 2 2020-03-11T22:46:57.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:46:57.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:46:57.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:46:57.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:46:57.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:46:57.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:46:57.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:46:57.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:46:57.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:46:57.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:46:57.553 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:46:57.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:46:57.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:46:57.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:46:57.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:46:57.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:46:57.554 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:46:57.554 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:46:57.554 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:46:57.555 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:46:57.555 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:46:57.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:46:57.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:46:57.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:46:57.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:46:57.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:46:57.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:46:57.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:46:57.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:46:57.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:46:57.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:46:57.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:46:57.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:46:57.557 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:46:57.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=97fe8809-de58-44f1-b42a-27146359a1d6 2020-03-11T22:46:57.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 97fe8809-de58-44f1-b42a-27146359a1d6' 2020-03-11T22:46:57.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:46:57.559 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 97fe8809-de58-44f1-b42a-27146359a1d6 2020-03-11T22:46:57.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBhamleYH2DIRAAiX1IL8d6xW8li2Tgf7SPoA== 2020-03-11T22:46:57.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBhamleYH2DIRAAiX1IL8d6xW8li2Tgf7SPoA=="}' 2020-03-11T22:46:57.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 97fe8809-de58-44f1-b42a-27146359a1d6 -i td/ec-error-rollforward/2/new.json 2020-03-11T22:46:58.051 INFO:tasks.workunit.client.0.smithi155.stdout:2 2020-03-11T22:46:58.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/2/new.json 2020-03-11T22:46:58.062 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBhamleYH2DIRAAiX1IL8d6xW8li2Tgf7SPoA== --osd-uuid 97fe8809-de58-44f1-b42a-27146359a1d6 2020-03-11T22:46:58.087 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:58.079+0000 7f79d5ecdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:58.087 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:58.082+0000 7f79d5ecdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:58.088 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:58.085+0000 7f79d5ecdec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:46:58.088 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:58.085+0000 7f79d5ecdec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:46:58.089 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:58.085+0000 7f79d5ecdec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:46:58.089 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:58.085+0000 7f79d5ecdec0 -1 bluestore(td/ec-error-rollforward/2/block) _read_bdev_label failed to open td/ec-error-rollforward/2/block: (2) No such file or directory 2020-03-11T22:46:58.185 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:46:58.183+0000 7f79d5ecdec0 -1 bluestore(td/ec-error-rollforward/2) _read_fsid unparsable uuid 2020-03-11T22:47:01.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/2/keyring 2020-03-11T22:47:01.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:01.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:47:01.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:01.231 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2020-03-11T22:47:01.683 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:47:01.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:47:01.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:01.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:01.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:01.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:01.701 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:01.697+0000 7ff43dceaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:01.736 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:01.732+0000 7ff43dceaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:01.738 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:01.734+0000 7ff43dceaec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:02.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:47:02.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:02.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:47:02.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:02.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:02.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:02.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:02.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:47:02.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:02.030 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:47:02.252 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:02.248+0000 7ff43dceaec0 -1 Falling back to public interface 2020-03-11T22:47:02.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:02.975 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:02.972+0000 7ff43dceaec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:47:03.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:03.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:03.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:03.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:03.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:47:03.371 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:47:03.716 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/54064,v1:127.0.0.1:6819/54064] [v2:127.0.0.1:6820/54064,v1:127.0.0.1:6821/54064] exists,up 97fe8809-de58-44f1-b42a-27146359a1d6 2020-03-11T22:47:03.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:03.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:03.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:03.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:33: TEST_ec_error_rollforward: run_osd td/ec-error-rollforward 3 2020-03-11T22:47:03.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/ec-error-rollforward 2020-03-11T22:47:03.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:47:03.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:47:03.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:47:03.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:47:03.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:47:03.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:03.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:03.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:03.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:47:03.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:47:03.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:03.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:47:03.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:47:03.719 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:47:03.719 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:03.719 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:03.720 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:03.720 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:03.720 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:47:03.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:47:03.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:47:03.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:03.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:47:03.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:47:03.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:47:03.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:47:03.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:03.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:03.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:03.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:47:03.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:47:03.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:47:03.722 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:47:03.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=6c37acfb-dbaa-4290-b100-4d3a296faab2 2020-03-11T22:47:03.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 6c37acfb-dbaa-4290-b100-4d3a296faab2' 2020-03-11T22:47:03.723 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:47:03.723 INFO:tasks.workunit.client.0.smithi155.stdout:add osd3 6c37acfb-dbaa-4290-b100-4d3a296faab2 2020-03-11T22:47:03.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBnamle35NjKxAAX1/7quMt+aVNIYWFTvFcAg== 2020-03-11T22:47:03.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBnamle35NjKxAAX1/7quMt+aVNIYWFTvFcAg=="}' 2020-03-11T22:47:03.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 6c37acfb-dbaa-4290-b100-4d3a296faab2 -i td/ec-error-rollforward/3/new.json 2020-03-11T22:47:04.189 INFO:tasks.workunit.client.0.smithi155.stdout:3 2020-03-11T22:47:04.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/ec-error-rollforward/3/new.json 2020-03-11T22:47:04.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBnamle35NjKxAAX1/7quMt+aVNIYWFTvFcAg== --osd-uuid 6c37acfb-dbaa-4290-b100-4d3a296faab2 2020-03-11T22:47:04.220 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:04.216+0000 7f35fcfacec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:04.222 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:04.218+0000 7f35fcfacec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:04.224 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:04.220+0000 7f35fcfacec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:04.224 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:04.221+0000 7f35fcfacec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:47:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:04.221+0000 7f35fcfacec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:47:04.225 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:04.221+0000 7f35fcfacec0 -1 bluestore(td/ec-error-rollforward/3/block) _read_bdev_label failed to open td/ec-error-rollforward/3/block: (2) No such file or directory 2020-03-11T22:47:04.282 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:04.279+0000 7f35fcfacec0 -1 bluestore(td/ec-error-rollforward/3) _read_fsid unparsable uuid 2020-03-11T22:47:07.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/ec-error-rollforward/3/keyring 2020-03-11T22:47:07.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:07.319 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd3 key to auth repository 2020-03-11T22:47:07.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:47:07.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/ec-error-rollforward/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:07.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:47:07.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:07.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:07.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:07.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:07.797 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.3 2020-03-11T22:47:07.814 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:07.811+0000 7f7e9096cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:07.816 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:07.813+0000 7f7e9096cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:07.818 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:07.814+0000 7f7e9096cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:08.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:47:08.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:08.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:47:08.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:08.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:08.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:08.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:08.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:08.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:47:08.141 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:47:08.334 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:08.330+0000 7f7e9096cec0 -1 Falling back to public interface 2020-03-11T22:47:08.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:09.114 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:09.110+0000 7f7e9096cec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:47:09.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:09.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:09.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:09.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:09.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:47:09.485 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:47:09.854 INFO:tasks.workunit.client.0.smithi155.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/54419,v1:127.0.0.1:6827/54419] [v2:127.0.0.1:6828/54419,v1:127.0.0.1:6829/54419] exists,up 6c37acfb-dbaa-4290-b100-4d3a296faab2 2020-03-11T22:47:09.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:09.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:09.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:09.855 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:35: TEST_ec_error_rollforward: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2020-03-11T22:47:10.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:36: TEST_ec_error_rollforward: ceph osd pool create ec 1 1 erasure ec-profile 2020-03-11T22:47:11.124 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'ec' already exists 2020-03-11T22:47:11.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:38: TEST_ec_error_rollforward: rados -p ec put foo /etc/passwd 2020-03-11T22:47:11.827 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: cat td/ec-error-rollforward/osd.2.pid 2020-03-11T22:47:11.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:40: TEST_ec_error_rollforward: kill -STOP 54064 2020-03-11T22:47:11.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:43: TEST_ec_error_rollforward: pids=54674 2020-03-11T22:47:11.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:44: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:47:11.828 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:42: TEST_ec_error_rollforward: rados -p ec rm foo 2020-03-11T22:47:12.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:46: TEST_ec_error_rollforward: pids+=' 54697' 2020-03-11T22:47:12.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:45: TEST_ec_error_rollforward: rados -p ec rm a 2020-03-11T22:47:12.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:48: TEST_ec_error_rollforward: pids+=' 54698' 2020-03-11T22:47:12.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:47: TEST_ec_error_rollforward: rados -p ec rm b 2020-03-11T22:47:12.830 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:50: TEST_ec_error_rollforward: pids+=' 54699' 2020-03-11T22:47:12.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:51: TEST_ec_error_rollforward: sleep 1 2020-03-11T22:47:12.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:49: TEST_ec_error_rollforward: rados -p ec rm c 2020-03-11T22:47:13.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:54: TEST_ec_error_rollforward: kill_daemons td/ec-error-rollforward KILL osd 2020-03-11T22:47:13.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:47:13.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:47:13.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:47:13.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:47:13.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:47:13.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:47:13.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:55: TEST_ec_error_rollforward: kill 54674 54697 54698 54699 2020-03-11T22:47:13.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:56: TEST_ec_error_rollforward: wait 2020-03-11T22:47:13.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:58: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 0 2020-03-11T22:47:13.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:47:13.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:47:13.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=0 2020-03-11T22:47:13.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:47:13.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/0 2020-03-11T22:47:13.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:47:13.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:13.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:13.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:13.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/0' 2020-03-11T22:47:13.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/0/journal' 2020-03-11T22:47:13.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:13.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:47:13.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:47:13.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:47:13.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:13.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:13.949 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:13.949 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:13.949 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:47:13.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:47:13.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:47:13.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:13.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:47:13.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:47:13.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:13.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:13.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:13.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:47:13.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:47:13.950 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/0 2020-03-11T22:47:13.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.0 2020-03-11T22:47:13.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 0 --fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/0 --osd-journal=td/ec-error-rollforward/0/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:13.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/0/whoami 2020-03-11T22:47:13.951 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:47:13.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 0 = 0 ']' 2020-03-11T22:47:13.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:47:13.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:47:13.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:47:13.966 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:13.962+0000 7f1346356ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:13.968 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:13.964+0000 7f1346356ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:13.969 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:13.965+0000 7f1346356ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:14.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 0 2020-03-11T22:47:14.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:14.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:47:14.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:14.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:14.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:14.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:14.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:14.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:47:14.313 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:47:14.484 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:14.480+0000 7f1346356ec0 -1 Falling back to public interface 2020-03-11T22:47:14.657 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/53326,v1:127.0.0.1:6803/53326] [v2:127.0.0.1:6804/53326,v1:127.0.0.1:6805/53326] exists,up 91d53f57-7a3f-4003-a1b6-9fcff03b9c24 2020-03-11T22:47:14.657 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:47:14.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:14.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:14.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:14.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:59: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 1 2020-03-11T22:47:14.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:47:14.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:47:14.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=1 2020-03-11T22:47:14.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:47:14.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/1 2020-03-11T22:47:14.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:47:14.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:14.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:14.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:14.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/1' 2020-03-11T22:47:14.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/1/journal' 2020-03-11T22:47:14.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:14.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:47:14.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:47:14.660 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:47:14.660 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:14.660 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:14.660 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:14.660 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:14.661 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:47:14.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:47:14.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:47:14.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:14.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:47:14.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:47:14.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:14.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:14.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:14.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:47:14.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:47:14.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/1 2020-03-11T22:47:14.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.1 2020-03-11T22:47:14.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 1 --fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/1 --osd-journal=td/ec-error-rollforward/1/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:14.663 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/1/whoami 2020-03-11T22:47:14.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 1 = 1 ']' 2020-03-11T22:47:14.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:47:14.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:47:14.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:47:14.673 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:14.669+0000 7fe71881eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:14.675 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:14.671+0000 7fe71881eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:14.676 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:14.673+0000 7fe71881eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:15.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 1 2020-03-11T22:47:15.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:15.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:47:15.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:15.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:15.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:15.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:15.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:15.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:47:15.156 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:47:15.192 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:15.188+0000 7fe71881eec0 -1 Falling back to public interface 2020-03-11T22:47:15.534 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 22 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/53681,v1:127.0.0.1:6811/53681] [v2:127.0.0.1:6812/53681,v1:127.0.0.1:6813/53681] exists,up 78203e12-07b3-4e1f-bc02-ed97fd0fcbb8 2020-03-11T22:47:15.534 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:47:15.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:15.535 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:15.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:15.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:60: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 2 2020-03-11T22:47:15.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:47:15.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:47:15.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=2 2020-03-11T22:47:15.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:47:15.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/2 2020-03-11T22:47:15.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:47:15.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:15.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:15.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:15.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/2' 2020-03-11T22:47:15.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/2/journal' 2020-03-11T22:47:15.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:15.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:47:15.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:47:15.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:47:15.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:15.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:15.539 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:15.539 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:15.539 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:47:15.539 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:47:15.539 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:47:15.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:15.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:47:15.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:47:15.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:15.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:15.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:15.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:47:15.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:47:15.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/2 2020-03-11T22:47:15.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.2 2020-03-11T22:47:15.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 2 --fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/2 --osd-journal=td/ec-error-rollforward/2/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:15.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/2/whoami 2020-03-11T22:47:15.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 2 = 2 ']' 2020-03-11T22:47:15.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:47:15.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:47:15.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:47:15.552 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:15.548+0000 7f7e657d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:15.555 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:15.551+0000 7f7e657d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:15.557 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:15.553+0000 7f7e657d0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:15.571 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:15.568+0000 7f1346356ec0 -1 osd.0 24 log_to_monitors {default=true} 2020-03-11T22:47:15.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 2 2020-03-11T22:47:15.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:15.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:47:15.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:15.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:15.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:15.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:15.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:15.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:47:15.942 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:47:16.073 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:16.068+0000 7f7e657d0ec0 -1 Falling back to public interface 2020-03-11T22:47:16.307 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:16.303+0000 7fe71881eec0 -1 osd.1 24 log_to_monitors {default=true} 2020-03-11T22:47:16.419 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/54064,v1:127.0.0.1:6819/54064] [v2:127.0.0.1:6820/54064,v1:127.0.0.1:6821/54064] exists,up 97fe8809-de58-44f1-b42a-27146359a1d6 2020-03-11T22:47:16.419 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.3 2020-03-11T22:47:16.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:16.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:16.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:16.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:61: TEST_ec_error_rollforward: activate_osd td/ec-error-rollforward 3 2020-03-11T22:47:16.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local dir=td/ec-error-rollforward 2020-03-11T22:47:16.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: shift 2020-03-11T22:47:16.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: local id=3 2020-03-11T22:47:16.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: shift 2020-03-11T22:47:16.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: local osd_data=td/ec-error-rollforward/3 2020-03-11T22:47:16.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: local 'ceph_args=--fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 ' 2020-03-11T22:47:16.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:16.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:16.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:16.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --osd-data=td/ec-error-rollforward/3' 2020-03-11T22:47:16.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --osd-journal=td/ec-error-rollforward/3/journal' 2020-03-11T22:47:16.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:16.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+= 2020-03-11T22:47:16.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --run-dir=td/ec-error-rollforward' 2020-03-11T22:47:16.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: get_asok_path 2020-03-11T22:47:16.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:16.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:16.422 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:16.422 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:16.422 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:47:16.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:47:16.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' 2020-03-11T22:47:16.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:16.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' --log-file=td/ec-error-rollforward/$name.log' 2020-03-11T22:47:16.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=' --pid-file=td/ec-error-rollforward/$name.pid' 2020-03-11T22:47:16.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:16.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:863: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:16.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:16.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph_args+=' ' 2020-03-11T22:47:16.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:866: activate_osd: ceph_args+= 2020-03-11T22:47:16.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: mkdir -p td/ec-error-rollforward/3 2020-03-11T22:47:16.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:869: activate_osd: echo start osd.3 2020-03-11T22:47:16.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph-osd -i 3 --fsid=859071d7-c176-461a-974d-f9a2e1c81fb1 --auth-supported=none --mon-host=127.0.0.1:7132 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ec-error-rollforward/3 --osd-journal=td/ec-error-rollforward/3/journal --chdir= --run-dir=td/ec-error-rollforward '--admin-socket=/tmp/ceph-asok.52897/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ec-error-rollforward/$name.log' '--pid-file=td/ec-error-rollforward/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:16.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: cat td/ec-error-rollforward/3/whoami 2020-03-11T22:47:16.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:872: activate_osd: '[' 3 = 3 ']' 2020-03-11T22:47:16.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: ceph osd dump --format=json 2020-03-11T22:47:16.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: jq '.flags_set[]' 2020-03-11T22:47:16.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:875: activate_osd: grep -q '"noup"' 2020-03-11T22:47:16.438 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:16.434+0000 7f840f6afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:16.449 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:16.445+0000 7f840f6afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:16.450 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:16.447+0000 7f840f6afec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:16.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:878: activate_osd: wait_for_osd up 3 2020-03-11T22:47:16.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:16.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:47:16.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:16.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:16.819 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:16.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:16.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:16.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:47:16.820 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:47:16.968 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:16.964+0000 7f840f6afec0 -1 Falling back to public interface 2020-03-11T22:47:17.218 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:17.310 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:17.307+0000 7f7e657d0ec0 -1 osd.2 23 log_to_monitors {default=true} 2020-03-11T22:47:18.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:18.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:18.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:18.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:18.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:47:18.221 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:47:18.281 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:18.278+0000 7f840f6afec0 -1 osd.3 24 log_to_monitors {default=true} 2020-03-11T22:47:18.626 INFO:tasks.workunit.client.0.smithi155.stdout:osd.3 up in weight 1 up_from 34 up_thru 0 down_at 27 last_clean_interval [19,24) [v2:127.0.0.1:6826/55249,v1:127.0.0.1:6827/55249] [v2:127.0.0.1:6828/55249,v1:127.0.0.1:6829/55249] exists,up 6c37acfb-dbaa-4290-b100-4d3a296faab2 2020-03-11T22:47:18.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:18.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:18.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:18.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:63: TEST_ec_error_rollforward: wait_for_clean 2020-03-11T22:47:18.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:47:18.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:47:18.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:47:18.628 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:47:18.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:47:18.629 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:47:18.629 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:47:18.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:47:18.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:47:18.629 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:47:18.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:47:18.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:47:18.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:47:18.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:47:18.717 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:47:19.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:47:19.056 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:47:19.056 INFO:tasks.workunit.client.0.smithi155.stderr:2 2020-03-11T22:47:19.056 INFO:tasks.workunit.client.0.smithi155.stderr:3' 2020-03-11T22:47:19.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:47:19.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:19.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:47:19.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=111669149699 2020-03-11T22:47:19.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-111669149699' 2020-03-11T22:47:19.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:19.178 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:47:19.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=124554051587 2020-03-11T22:47:19.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-111669149699 1-124554051587' 2020-03-11T22:47:19.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:19.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:47:19.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=137438953475 2020-03-11T22:47:19.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-111669149699 1-124554051587 2-137438953475' 2020-03-11T22:47:19.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:19.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:47:19.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=146028888067 2020-03-11T22:47:19.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-111669149699 1-124554051587 2-137438953475 3-146028888067' 2020-03-11T22:47:19.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:19.568 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-111669149699 2020-03-11T22:47:19.568 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:19.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:47:19.570 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-111669149699 2020-03-11T22:47:19.570 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:19.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=111669149699 2020-03-11T22:47:19.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 111669149699' 2020-03-11T22:47:19.571 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 111669149699 2020-03-11T22:47:19.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:19.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 111669149699 -lt 111669149699 2020-03-11T22:47:19.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:19.933 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-124554051587 2020-03-11T22:47:19.933 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:19.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:47:19.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-124554051587 2020-03-11T22:47:19.935 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:19.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=124554051587 2020-03-11T22:47:19.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 124554051587' 2020-03-11T22:47:19.936 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 124554051587 2020-03-11T22:47:19.937 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:47:20.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 124554051587 -lt 124554051587 2020-03-11T22:47:20.281 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:20.282 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-137438953475 2020-03-11T22:47:20.282 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:20.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:47:20.284 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-137438953475 2020-03-11T22:47:20.284 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:20.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=137438953475 2020-03-11T22:47:20.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 137438953475' 2020-03-11T22:47:20.285 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 137438953475 2020-03-11T22:47:20.285 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:47:20.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 1 -lt 137438953475 2020-03-11T22:47:20.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:47:21.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:47:21.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:47:21.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 137438953475 -lt 137438953475 2020-03-11T22:47:21.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:21.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-146028888067 2020-03-11T22:47:21.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:21.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:47:21.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-146028888067 2020-03-11T22:47:21.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:21.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=146028888067 2020-03-11T22:47:21.980 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 146028888067' 2020-03-11T22:47:21.980 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 146028888067 2020-03-11T22:47:21.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:47:22.323 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 146028888067 -lt 146028888067 2020-03-11T22:47:22.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:47:22.324 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:22.324 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:22.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:47:22.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:22.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:22.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:22.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:22.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:22.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:22.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:23.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:47:23.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:23.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:23.089 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:23.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:47:23.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:47:23.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:47:23.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:47:23.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:47:23.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:47:23.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:47:23.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:23.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:23.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:23.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:23.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:23.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:23.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:24.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:47:24.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:24.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:24.018 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:24.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:47:24.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:47:24.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:47:24.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/ec-error-rollforward.sh:22: run: teardown td/ec-error-rollforward 2020-03-11T22:47:24.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:47:24.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:47:24.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:47:24.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:47:24.459 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:47:24.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:47:24.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:47:24.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:47:24.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:47:24.566 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:47:24.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:47:24.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:47:24.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:47:24.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:47:24.568 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:47:24.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:47:24.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:47:24.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:47:24.571 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:47:24.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:47:24.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:47:24.572 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:47:24.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:47:24.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:24.578 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:47:24.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52897 2020-03-11T22:47:24.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:47:24.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:47:24.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:47:24.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/ec-error-rollforward 0 2020-03-11T22:47:24.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/ec-error-rollforward 2020-03-11T22:47:24.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:47:24.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/ec-error-rollforward KILL 2020-03-11T22:47:24.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:47:24.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:47:24.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:47:24.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:47:24.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:47:24.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:47:24.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:47:24.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:47:24.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:47:24.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:47:24.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:47:24.583 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:47:24.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:47:24.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:47:24.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:47:24.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:47:24.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:47:24.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:47:24.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/ec-error-rollforward 2020-03-11T22:47:24.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:47:24.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:24.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.52897 2020-03-11T22:47:24.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.52897 2020-03-11T22:47:24.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:47:24.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:47:24.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:47:24.590 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:47:24.590 INFO:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:47:24.664 INFO:tasks.workunit:Running workunit osd/osd-backfill-prio.sh... 2020-03-11T22:47:24.665 INFO:teuthology.orchestra.run.smithi155:workunit test osd/osd-backfill-prio.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh 2020-03-11T22:47:24.715 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:47:24.725 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:47:24.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:47:24.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:47:24.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:47:24.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:47:24.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:47:24.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:47:24.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:47:24.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:47:24.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-prio 2020-03-11T22:47:24.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:21: run: local dir=td/osd-backfill-prio 2020-03-11T22:47:24.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:22: run: shift 2020-03-11T22:47:24.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: export CEPH_MON=127.0.0.1:7114 2020-03-11T22:47:24.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:25: run: CEPH_MON=127.0.0.1:7114 2020-03-11T22:47:24.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:26: run: export CEPH_ARGS 2020-03-11T22:47:24.727 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: uuidgen 2020-03-11T22:47:24.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:27: run: CEPH_ARGS+='--fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none ' 2020-03-11T22:47:24.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:47:24.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:29: run: CEPH_ARGS+='--osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:47:24.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: export objects=50 2020-03-11T22:47:24.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:30: run: objects=50 2020-03-11T22:47:24.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: export poolprefix=test 2020-03-11T22:47:24.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:31: run: poolprefix=test 2020-03-11T22:47:24.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: export FORCE_PRIO=254 2020-03-11T22:47:24.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:32: run: FORCE_PRIO=254 2020-03-11T22:47:24.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: export DEGRADED_PRIO=150 2020-03-11T22:47:24.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:33: run: DEGRADED_PRIO=150 2020-03-11T22:47:24.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: export NORMAL_PRIO=110 2020-03-11T22:47:24.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:34: run: NORMAL_PRIO=110 2020-03-11T22:47:24.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: set 2020-03-11T22:47:24.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:47:24.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:36: run: local 'funcs=TEST_backfill_pool_priority 2020-03-11T22:47:24.732 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_backfill_priority' 2020-03-11T22:47:24.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:47:24.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:47:24.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:47:24.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:47:24.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:47:24.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:47:24.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:47:24.733 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:47:24.733 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:47:24.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:47:24.733 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:47:24.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:47:24.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:47:24.734 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:47:24.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:47:24.734 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:47:24.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:47:24.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:47:24.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:47:24.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:47:24.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:47:24.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:47:24.735 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:47:24.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:47:24.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:47:24.737 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:47:24.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:47:24.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:24.737 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:47:24.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56110 2020-03-11T22:47:24.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:47:24.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:47:24.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:47:24.738 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:47:24.738 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:24.738 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:47:24.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.56110 2020-03-11T22:47:24.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:47:24.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:47:24.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:47:24.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:47:24.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:47:24.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_pool_priority td/osd-backfill-prio 2020-03-11T22:47:24.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:326: TEST_backfill_pool_priority: local dir=td/osd-backfill-prio 2020-03-11T22:47:24.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:327: TEST_backfill_pool_priority: local pools=3 2020-03-11T22:47:24.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:328: TEST_backfill_pool_priority: local OSDS=2 2020-03-11T22:47:24.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:330: TEST_backfill_pool_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:47:24.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:47:24.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:47:24.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:47:24.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:47:24.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:47:24.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:47:25.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:47:25.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:25.060 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:25.061 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:25.061 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:25.061 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:47:25.061 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:47:25.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:47:25.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:47:25.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:47:25.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:47:25.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:47:25.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:47:25.205 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:47:25.205 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:47:25.206 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:47:25.206 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:47:25.206 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:25.206 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:25.206 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:47:25.206 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-mon.a.asok 2020-03-11T22:47:25.206 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:47:25.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56110/ceph-mon.a.asok config get fsid 2020-03-11T22:47:25.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:47:25.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:47:25.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:47:25.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:47:25.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:47:25.293 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:47:25.293 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:47:25.293 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:47:25.293 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:47:25.294 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:25.294 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:47:25.294 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-mon.a.asok 2020-03-11T22:47:25.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:47:25.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56110/ceph-mon.a.asok config get mon_host 2020-03-11T22:47:25.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:331: TEST_backfill_pool_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:47:25.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:47:25.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:47:25.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:47:25.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:47:25.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:47:25.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:47:26.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:47:26.261 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:26.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:26.262 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:26.262 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:26.262 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:47:26.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:47:26.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:47:26.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:47:26.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:332: TEST_backfill_pool_priority: export CEPH_ARGS 2020-03-11T22:47:26.285 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: expr 2 - 1 2020-03-11T22:47:26.286 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: seq 0 1 2020-03-11T22:47:26.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:47:26.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:47:26.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:47:26.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:47:26.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:47:26.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:47:26.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:47:26.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:47:26.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:26.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:26.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:26.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:47:26.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:47:26.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:26.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:47:26.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:47:26.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:47:26.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:26.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:26.293 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:26.293 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:26.293 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:47:26.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:47:26.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:47:26.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:26.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:47:26.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:47:26.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:47:26.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:47:26.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:26.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:26.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:26.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:47:26.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:47:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:47:26.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:47:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=b0b73cb2-ec87-4bb5-9f93-08b675dd5ca5 2020-03-11T22:47:26.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 b0b73cb2-ec87-4bb5-9f93-08b675dd5ca5' 2020-03-11T22:47:26.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:47:26.295 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 b0b73cb2-ec87-4bb5-9f93-08b675dd5ca5 2020-03-11T22:47:26.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQB+amleSoQUEhAAfEL22BTEITXb5BAHpNNNWw== 2020-03-11T22:47:26.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQB+amleSoQUEhAAfEL22BTEITXb5BAHpNNNWw=="}' 2020-03-11T22:47:26.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new b0b73cb2-ec87-4bb5-9f93-08b675dd5ca5 -i td/osd-backfill-prio/0/new.json 2020-03-11T22:47:26.643 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:47:26.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:47:26.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQB+amleSoQUEhAAfEL22BTEITXb5BAHpNNNWw== --osd-uuid b0b73cb2-ec87-4bb5-9f93-08b675dd5ca5 2020-03-11T22:47:26.678 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:26.676+0000 7fcd1de62ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:26.680 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:26.678+0000 7fcd1de62ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:26.682 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:26.679+0000 7fcd1de62ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:26.682 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:26.680+0000 7fcd1de62ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:47:26.683 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:26.680+0000 7fcd1de62ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:47:26.683 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:26.680+0000 7fcd1de62ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:47:26.727 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:26.724+0000 7fcd1de62ec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:47:29.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:47:29.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:29.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:47:29.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:29.743 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2020-03-11T22:47:30.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:47:30.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:30.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:30.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:30.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:30.194 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:47:30.217 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:30.214+0000 7f2dff3f2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:30.228 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:30.225+0000 7f2dff3f2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:30.228 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:30.226+0000 7f2dff3f2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:30.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:47:30.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:30.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:47:30.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:30.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:30.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:30.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:30.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:30.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:47:30.542 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:47:30.744 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:30.741+0000 7f2dff3f2ec0 -1 Falling back to public interface 2020-03-11T22:47:30.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:31.470 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:31.468+0000 7f2dff3f2ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:47:31.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:31.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:31.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:31.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:31.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:47:31.888 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:47:32.301 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/56542,v1:127.0.0.1:6803/56542] [v2:127.0.0.1:6804/56542,v1:127.0.0.1:6805/56542] exists,up b0b73cb2-ec87-4bb5-9f93-08b675dd5ca5 2020-03-11T22:47:32.301 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 556b8f83-acee-4f72-b656-0c5ebb5f662b 2020-03-11T22:47:32.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:32.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:32.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:32.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:334: TEST_backfill_pool_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:47:32.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:336: TEST_backfill_pool_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:47:32.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:47:32.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:47:32.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:47:32.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:47:32.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:47:32.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:47:32.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:47:32.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:47:32.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:47:32.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:47:32.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:47:32.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:47:32.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:47:32.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:47:32.304 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:47:32.304 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:47:32.304 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:47:32.304 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:47:32.305 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:47:32.305 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:47:32.305 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:47:32.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:47:32.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:47:32.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:47:32.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:47:32.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:47:32.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:47:32.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:47:32.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:47:32.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:32.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:47:32.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:47:32.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:47:32.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:47:32.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=556b8f83-acee-4f72-b656-0c5ebb5f662b 2020-03-11T22:47:32.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 556b8f83-acee-4f72-b656-0c5ebb5f662b' 2020-03-11T22:47:32.307 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:47:32.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQCEamlenhq7EhAAEoNRZFx7minu3Bup3FT/0Q== 2020-03-11T22:47:32.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQCEamlenhq7EhAAEoNRZFx7minu3Bup3FT/0Q=="}' 2020-03-11T22:47:32.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 556b8f83-acee-4f72-b656-0c5ebb5f662b -i td/osd-backfill-prio/1/new.json 2020-03-11T22:47:32.800 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:47:32.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:47:32.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQCEamlenhq7EhAAEoNRZFx7minu3Bup3FT/0Q== --osd-uuid 556b8f83-acee-4f72-b656-0c5ebb5f662b 2020-03-11T22:47:32.836 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:32.833+0000 7f86447ffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:32.860 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:32.858+0000 7f86447ffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:32.862 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:32.859+0000 7f86447ffec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:32.862 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:32.860+0000 7f86447ffec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:47:32.862 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:32.860+0000 7f86447ffec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:47:32.863 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:32.860+0000 7f86447ffec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:47:32.960 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:32.958+0000 7f86447ffec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:47:35.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:47:35.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:47:35.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:47:35.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:47:35.987 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2020-03-11T22:47:36.450 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:47:36.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:47:36.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:47:36.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:47:36.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:47:36.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:47:36.474 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:36.471+0000 7f7fadcd9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:36.475 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:36.473+0000 7f7fadcd9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:36.477 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:36.474+0000 7f7fadcd9ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:47:36.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:47:36.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:47:36.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:47:36.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:47:36.798 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:47:36.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:36.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:47:36.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:36.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:47:36.799 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:47:36.992 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:36.989+0000 7f7fadcd9ec0 -1 Falling back to public interface 2020-03-11T22:47:37.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:47:37.791 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:47:37.789+0000 7f7fadcd9ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:47:38.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:47:38.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:47:38.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:47:38.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:47:38.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:47:38.147 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:47:38.517 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/56897,v1:127.0.0.1:6811/56897] [v2:127.0.0.1:6812/56897,v1:127.0.0.1:6813/56897] exists,up 556b8f83-acee-4f72-b656-0c5ebb5f662b 2020-03-11T22:47:38.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:47:38.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:47:38.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:47:38.518 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:47:38.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:38.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test1 1 1 2020-03-11T22:47:38.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:47:39.100 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test1' already exists 2020-03-11T22:47:39.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:40.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:47:41.004 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 size to 2 2020-03-11T22:47:41.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:41.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test2 1 1 2020-03-11T22:47:41.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:47:41.555 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test2' already exists 2020-03-11T22:47:41.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:42.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:47:43.324 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 2 size to 2 2020-03-11T22:47:43.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:339: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:43.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:341: TEST_backfill_pool_priority: create_pool test3 1 1 2020-03-11T22:47:43.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:47:43.864 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test3' already exists 2020-03-11T22:47:43.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:47:44.873 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:342: TEST_backfill_pool_priority: ceph osd pool set test3 size 2 2020-03-11T22:47:45.570 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 3 size to 2 2020-03-11T22:47:45.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:344: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:47:50.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:346: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:47:50.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:47:50.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:47:50.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:47:50.580 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:47:50.581 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:47:50.581 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:47:50.582 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:47:50.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:47:50.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:47:50.582 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:47:50.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:47:50.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:47:50.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:47:50.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:47:50.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:47:51.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:47:51.038 INFO:tasks.workunit.client.0.smithi155.stderr:1' 2020-03-11T22:47:51.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:47:51.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:51.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:47:51.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836486 2020-03-11T22:47:51.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486' 2020-03-11T22:47:51.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:51.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:47:51.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672965 2020-03-11T22:47:51.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836486 1-42949672965' 2020-03-11T22:47:51.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:51.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836486 2020-03-11T22:47:51.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:51.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:47:51.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836486 2020-03-11T22:47:51.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:51.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836486 2020-03-11T22:47:51.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2020-03-11T22:47:51.297 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836486 2020-03-11T22:47:51.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:51.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836484 -lt 21474836486 2020-03-11T22:47:51.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:47:52.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:47:52.645 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:53.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836486 -lt 21474836486 2020-03-11T22:47:53.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:53.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672965 2020-03-11T22:47:53.021 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:53.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:47:53.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672965 2020-03-11T22:47:53.023 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:53.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672965 2020-03-11T22:47:53.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672965' 2020-03-11T22:47:53.024 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672965 2020-03-11T22:47:53.024 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:47:53.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672965 -lt 42949672965 2020-03-11T22:47:53.371 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:47:53.371 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:53.371 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:53.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:47:53.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:47:53.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:47:53.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:47:53.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:47:53.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:47:53.850 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:47:53.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:47:54.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:47:54.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:47:54.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:47:54.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:47:54.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:47:54.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:47:54.644 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:47:54.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:348: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:47:54.966 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:47:54.966 INFO:tasks.workunit.client.0.smithi155.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:46.575467+0000 0'0 28:18 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:47:43.662176+0000 0'0 2020-03-11T22:47:43.662176+0000 0 2020-03-11T22:47:54.967 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:40.903886+0000 0'0 27:29 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:47:38.895682+0000 0'0 2020-03-11T22:47:38.895682+0000 0 2020-03-11T22:47:54.967 INFO:tasks.workunit.client.0.smithi155.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:47:43.728976+0000 0'0 26:21 [0,1] 0 [0,1] 0 0'0 2020-03-11T22:47:41.316437+0000 0'0 2020-03-11T22:47:41.316437+0000 0 2020-03-11T22:47:54.967 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:47:54.967 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:47:54.968 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:47:54.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:352: TEST_backfill_pool_priority: local PG1 2020-03-11T22:47:54.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:353: TEST_backfill_pool_priority: local POOLNUM1 2020-03-11T22:47:54.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:354: TEST_backfill_pool_priority: local pool1 2020-03-11T22:47:54.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:355: TEST_backfill_pool_priority: local chk_osd1_1 2020-03-11T22:47:54.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:356: TEST_backfill_pool_priority: local chk_osd1_2 2020-03-11T22:47:54.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:358: TEST_backfill_pool_priority: local PG2 2020-03-11T22:47:54.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:359: TEST_backfill_pool_priority: local POOLNUM2 2020-03-11T22:47:54.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:360: TEST_backfill_pool_priority: local pool2 2020-03-11T22:47:54.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:361: TEST_backfill_pool_priority: local chk_osd2_1 2020-03-11T22:47:54.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:362: TEST_backfill_pool_priority: local chk_osd2_2 2020-03-11T22:47:54.978 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:47:54.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:54.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 1.0 --format=json 2020-03-11T22:47:54.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:47:55.315 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:47:55.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=1 2020-03-11T22:47:55.317 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:47:55.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=0 2020-03-11T22:47:55.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z '' ']' 2020-03-11T22:47:55.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:371: TEST_backfill_pool_priority: PG1=1.0 2020-03-11T22:47:55.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:372: TEST_backfill_pool_priority: POOLNUM1=1 2020-03-11T22:47:55.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:373: TEST_backfill_pool_priority: pool1=test1 2020-03-11T22:47:55.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:374: TEST_backfill_pool_priority: chk_osd1_1=1 2020-03-11T22:47:55.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:375: TEST_backfill_pool_priority: chk_osd1_2=0 2020-03-11T22:47:55.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:364: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:55.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: ceph pg map 2.0 --format=json 2020-03-11T22:47:55.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:366: TEST_backfill_pool_priority: jq '.acting[]' 2020-03-11T22:47:55.657 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:47:55.657 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:367: TEST_backfill_pool_priority: local test_osd1=0 2020-03-11T22:47:55.658 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:47:55.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:368: TEST_backfill_pool_priority: local test_osd2=1 2020-03-11T22:47:55.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:369: TEST_backfill_pool_priority: '[' -z 1.0 ']' 2020-03-11T22:47:55.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:376: TEST_backfill_pool_priority: '[' 1 '!=' 0 ']' 2020-03-11T22:47:55.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:378: TEST_backfill_pool_priority: PG2=2.0 2020-03-11T22:47:55.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:379: TEST_backfill_pool_priority: POOLNUM2=2 2020-03-11T22:47:55.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:380: TEST_backfill_pool_priority: pool2=test2 2020-03-11T22:47:55.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:381: TEST_backfill_pool_priority: chk_osd2_1=0 2020-03-11T22:47:55.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:382: TEST_backfill_pool_priority: chk_osd2_2=1 2020-03-11T22:47:55.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:383: TEST_backfill_pool_priority: break 2020-03-11T22:47:55.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:386: TEST_backfill_pool_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:47:55.670 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:388: TEST_backfill_pool_priority: '[' test2 = '' ']' 2020-03-11T22:47:55.671 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: seq 1 3 2020-03-11T22:47:55.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:55.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 1 '!=' 1 -a 1 '!=' 2 ']' 2020-03-11T22:47:55.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:55.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 2 '!=' 1 -a 2 '!=' 2 ']' 2020-03-11T22:47:55.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:394: TEST_backfill_pool_priority: for p in $(seq 1 $pools) 2020-03-11T22:47:55.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:396: TEST_backfill_pool_priority: '[' 3 '!=' 1 -a 3 '!=' 2 ']' 2020-03-11T22:47:55.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:398: TEST_backfill_pool_priority: delete_pool test3 2020-03-11T22:47:55.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:47:55.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:47:56.179 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test3' does not exist 2020-03-11T22:47:56.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:402: TEST_backfill_pool_priority: pool1_extra_prio=1 2020-03-11T22:47:56.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:403: TEST_backfill_pool_priority: pool2_extra_prio=2 2020-03-11T22:47:56.187 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: expr 150 + 1 + 1 2020-03-11T22:47:56.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:405: TEST_backfill_pool_priority: pool1_prio=152 2020-03-11T22:47:56.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: expr 150 + 1 + 2 2020-03-11T22:47:56.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:406: TEST_backfill_pool_priority: pool2_prio=153 2020-03-11T22:47:56.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:408: TEST_backfill_pool_priority: ceph osd pool set test1 size 1 2020-03-11T22:47:56.834 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 size to 1 2020-03-11T22:47:56.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:409: TEST_backfill_pool_priority: ceph osd pool set test1 recovery_priority 1 2020-03-11T22:47:57.502 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 recovery_priority to 1 2020-03-11T22:47:57.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:410: TEST_backfill_pool_priority: ceph osd pool set test2 size 1 2020-03-11T22:47:58.095 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 2 size to 1 2020-03-11T22:47:58.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:411: TEST_backfill_pool_priority: ceph osd pool set test2 recovery_priority 2 2020-03-11T22:47:58.671 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 2 recovery_priority to 2 2020-03-11T22:47:58.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:412: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:47:58.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:47:58.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:47:58.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:47:58.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:47:58.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:47:58.682 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:47:58.682 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:47:58.682 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:47:58.682 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:47:58.682 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:47:58.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:47:58.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:47:58.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:47:58.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:47:58.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:47:59.124 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:47:59.125 INFO:tasks.workunit.client.0.smithi155.stderr:1' 2020-03-11T22:47:59.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:47:59.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:59.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:47:59.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836490 2020-03-11T22:47:59.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490' 2020-03-11T22:47:59.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:47:59.294 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:47:59.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672969 2020-03-11T22:47:59.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836490 1-42949672969' 2020-03-11T22:47:59.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:59.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836490 2020-03-11T22:47:59.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:59.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:47:59.423 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836490 2020-03-11T22:47:59.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:59.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836490 2020-03-11T22:47:59.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2020-03-11T22:47:59.425 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836490 2020-03-11T22:47:59.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:47:59.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836490 -lt 21474836490 2020-03-11T22:47:59.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:47:59.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672969 2020-03-11T22:47:59.772 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:47:59.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:47:59.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672969 2020-03-11T22:47:59.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:47:59.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672969 2020-03-11T22:47:59.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672969' 2020-03-11T22:47:59.775 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672969 2020-03-11T22:47:59.775 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:48:00.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672969 -lt 42949672969 2020-03-11T22:48:00.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:48:00.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:00.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:00.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:48:00.612 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:00.612 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:00.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:00.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:00.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:00.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:00.613 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:00.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:48:00.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:00.945 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:00.946 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:01.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:48:01.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:48:01.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:48:01.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:414: TEST_backfill_pool_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:48:01.436 INFO:tasks.workunit.client.0.smithi155.stderr:10+0 records in 2020-03-11T22:48:01.436 INFO:tasks.workunit.client.0.smithi155.stderr:10+0 records out 2020-03-11T22:48:01.436 INFO:tasks.workunit.client.0.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0474892 s, 221 MB/s 2020-03-11T22:48:01.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:415: TEST_backfill_pool_priority: p=1 2020-03-11T22:48:01.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:48:01.437 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:48:01.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:01.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:48:01.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:01.706 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:48:01.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:01.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:48:02.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:02.223 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:48:02.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:02.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:48:02.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:02.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:48:02.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:02.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:48:03.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:03.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:48:03.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:03.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:48:03.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:03.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:48:03.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:03.974 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:48:04.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:04.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:48:04.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:04.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:48:04.715 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:04.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:48:04.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:04.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:48:05.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:05.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:48:05.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:05.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:48:05.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:05.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:48:06.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:06.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:48:06.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:06.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:48:06.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:06.734 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:48:06.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:06.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:48:07.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:07.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:48:07.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:07.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:48:07.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:07.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:48:08.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:08.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:48:08.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:08.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:48:08.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:08.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:48:08.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:08.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:48:09.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:09.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:48:09.359 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:09.360 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:48:09.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:09.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:48:09.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:09.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:48:10.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:10.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:48:10.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:10.509 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:48:10.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:10.784 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:48:11.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:11.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:48:11.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:11.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:48:11.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:11.544 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:48:11.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:11.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:48:12.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:12.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:48:12.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:12.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:48:12.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:12.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:48:12.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:12.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:48:13.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:13.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:48:13.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:13.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:48:13.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:13.744 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:48:14.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:14.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:48:14.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:14.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:48:14.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:14.545 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:48:14.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 1 + 1 2020-03-11T22:48:14.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=2 2020-03-11T22:48:14.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:416: TEST_backfill_pool_priority: for pname in $pool1 $pool2 2020-03-11T22:48:14.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: seq 1 50 2020-03-11T22:48:14.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:14.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:48:15.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:15.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:48:15.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:15.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:48:15.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:15.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:48:16.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:16.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:48:16.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:16.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:48:16.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:16.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:48:16.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:16.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:48:17.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:17.110 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:48:17.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:17.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:48:17.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:17.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:48:17.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:17.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:48:18.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:18.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:48:18.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:18.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:48:18.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:18.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:48:19.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:19.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:48:19.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:19.260 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:48:19.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:19.502 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:48:19.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:19.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:48:20.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:20.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:48:20.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:20.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:48:20.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:20.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:48:20.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:20.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:48:21.119 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:21.120 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:48:21.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:21.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:48:21.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:21.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:48:21.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:21.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:48:22.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:22.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:48:22.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:22.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:48:22.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:22.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:48:23.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:23.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:48:23.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:23.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:48:23.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:23.611 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:48:23.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:23.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:48:24.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:24.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:48:24.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:24.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:48:24.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:24.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:48:24.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:24.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:48:25.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:25.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:48:25.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:25.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:48:25.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:25.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:48:26.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:26.133 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:48:26.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:26.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:48:26.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:26.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:48:26.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:26.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:48:27.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:27.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:48:27.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:27.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:48:27.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:27.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:48:28.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:28.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:48:28.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:418: TEST_backfill_pool_priority: for i in $(seq 1 $objects) 2020-03-11T22:48:28.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:420: TEST_backfill_pool_priority: rados -p test2 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:48:28.598 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: expr 2 + 1 2020-03-11T22:48:28.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:422: TEST_backfill_pool_priority: p=3 2020-03-11T22:48:28.599 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: get_not_primary test1 obj1-p1 2020-03-11T22:48:28.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:48:28.600 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:48:28.601 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:48:28.601 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:48:28.601 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:48:28.601 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:48:28.601 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:48:28.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:48:28.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:48:28.953 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:48:29.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:425: TEST_backfill_pool_priority: local otherosd=null 2020-03-11T22:48:29.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:427: TEST_backfill_pool_priority: ceph pg dump pgs 2020-03-11T22:48:29.633 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:48:29.634 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:47:56.861100+0000 37'150 37:193 [1] 1 [1] 1 0'0 2020-03-11T22:47:38.895682+0000 0'0 2020-03-11T22:47:38.895682+0000 0 2020-03-11T22:48:29.634 INFO:tasks.workunit.client.0.smithi155.stdout:2.0 47 0 0 0 0 486539264 0 0 39 39 active+clean 2020-03-11T22:47:58.132419+0000 37'139 37:174 [0] 0 [0] 0 0'0 2020-03-11T22:47:41.316437+0000 0'0 2020-03-11T22:47:41.316437+0000 0 2020-03-11T22:48:29.634 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:48:29.634 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:48:29.635 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:48:29.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:428: TEST_backfill_pool_priority: ERRORS=0 2020-03-11T22:48:29.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:430: TEST_backfill_pool_priority: ceph osd pool set test1 size 2 2020-03-11T22:48:30.340 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 size to 2 2020-03-11T22:48:30.349 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:431: TEST_backfill_pool_priority: ceph osd pool set test2 size 2 2020-03-11T22:48:30.958 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 2 size to 2 2020-03-11T22:48:30.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:432: TEST_backfill_pool_priority: sleep 5 2020-03-11T22:48:35.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: get_asok_path osd.1 2020-03-11T22:48:35.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:48:35.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:48:35.969 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:35.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:35.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:48:35.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-osd.1.asok 2020-03-11T22:48:35.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:48:35.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:433: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:48:36.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:434: TEST_backfill_pool_priority: echo osd.1 2020-03-11T22:48:36.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:435: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:48:36.077 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 2020-03-11T22:48:36.078 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:48:36.078 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:48:36.078 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:48:36.078 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:48:36.078 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:48:36.078 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:48:36.078 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:48:36.078 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:48:36.079 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 152, 2020-03-11T22:48:36.079 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:48:36.079 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:48:36.079 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:48:36.079 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:48:36.079 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:48:36.079 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:48:36.079 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:48:36.080 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:48:36.080 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:48:36.080 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:48:36.080 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "2.0", 2020-03-11T22:48:36.080 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 153, 2020-03-11T22:48:36.080 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:48:36.080 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:48:36.080 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:48:36.081 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:48:36.081 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:48:36.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: get_asok_path osd.0 2020-03-11T22:48:36.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.0 2020-03-11T22:48:36.082 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.0 ']' 2020-03-11T22:48:36.082 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:48:36.082 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:48:36.082 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:48:36.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-osd.0.asok 2020-03-11T22:48:36.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: CEPH_ARGS= 2020-03-11T22:48:36.083 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:436: TEST_backfill_pool_priority: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.0.asok dump_recovery_reservations 2020-03-11T22:48:36.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:437: TEST_backfill_pool_priority: echo osd.0 2020-03-11T22:48:36.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:438: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:48:36.178 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 2020-03-11T22:48:36.178 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:48:36.178 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:48:36.179 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:48:36.179 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:48:36.179 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:48:36.179 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:48:36.179 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:48:36.179 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "2.0", 2020-03-11T22:48:36.179 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 153, 2020-03-11T22:48:36.179 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:48:36.180 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:48:36.180 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:48:36.180 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:48:36.180 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:48:36.180 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:48:36.180 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:48:36.180 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:48:36.181 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:48:36.181 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:48:36.181 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:48:36.181 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 152, 2020-03-11T22:48:36.181 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:48:36.181 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:48:36.181 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:48:36.181 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:48:36.182 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:48:36.182 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:48:36.183 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:48:36.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:48:36.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:441: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:48:36.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:442: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:48:36.185 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:48:36.185 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:48:36.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:447: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:48:36.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:448: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:48:36.186 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:48:36.186 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:48:36.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: eval 'ITEM="1.0"' 2020-03-11T22:48:36.188 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:456: TEST_backfill_pool_priority: ITEM=1.0 2020-03-11T22:48:36.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:457: TEST_backfill_pool_priority: '[' 1.0 '!=' 1.0 ']' 2020-03-11T22:48:36.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:48:36.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:48:36.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:462: TEST_backfill_pool_priority: PRIO=152 2020-03-11T22:48:36.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:463: TEST_backfill_pool_priority: '[' 152 '!=' 152 ']' 2020-03-11T22:48:36.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:48:36.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:48:36.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:48:36.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:471: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:48:36.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:472: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:48:36.195 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.0.out 2020-03-11T22:48:36.195 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:48:36.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:477: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:48:36.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:478: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:48:36.198 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:48:36.199 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].item' 2020-03-11T22:48:36.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: eval 'ITEM="2.0"' 2020-03-11T22:48:36.201 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:486: TEST_backfill_pool_priority: ITEM=2.0 2020-03-11T22:48:36.201 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:487: TEST_backfill_pool_priority: '[' 2.0 '!=' 2.0 ']' 2020-03-11T22:48:36.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: cat td/osd-backfill-prio/dump.1.out 2020-03-11T22:48:36.202 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: jq '.remote_reservations.in_progress[0].prio' 2020-03-11T22:48:36.204 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:492: TEST_backfill_pool_priority: PRIO=153 2020-03-11T22:48:36.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:493: TEST_backfill_pool_priority: '[' 153 '!=' 153 ']' 2020-03-11T22:48:36.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:500: TEST_backfill_pool_priority: wait_for_clean 2020-03-11T22:48:36.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:48:36.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:48:36.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:48:36.205 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:48:36.206 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:48:36.206 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:48:36.206 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:48:36.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:48:36.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:48:36.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:48:36.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:48:36.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:48:36.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:48:36.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:48:36.296 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:48:36.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:48:36.647 INFO:tasks.workunit.client.0.smithi155.stderr:1' 2020-03-11T22:48:36.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:48:36.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:36.648 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:48:36.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836499 2020-03-11T22:48:36.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499' 2020-03-11T22:48:36.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:48:36.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:48:37.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=42949672978 2020-03-11T22:48:37.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836499 1-42949672978' 2020-03-11T22:48:37.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:37.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836499 2020-03-11T22:48:37.112 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:37.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:48:37.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836499 2020-03-11T22:48:37.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:37.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836499 2020-03-11T22:48:37.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836499' 2020-03-11T22:48:37.115 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836499 2020-03-11T22:48:37.115 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:37.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836497 -lt 21474836499 2020-03-11T22:48:37.577 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:48:38.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:48:38.579 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:48:38.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836499 -lt 21474836499 2020-03-11T22:48:38.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:48:38.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-42949672978 2020-03-11T22:48:38.934 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:48:38.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:48:38.937 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-42949672978 2020-03-11T22:48:38.937 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:48:38.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=42949672978 2020-03-11T22:48:38.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 42949672978' 2020-03-11T22:48:38.937 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 42949672978 2020-03-11T22:48:38.938 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:48:39.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 42949672978 -lt 42949672978 2020-03-11T22:48:39.403 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:48:39.404 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:39.404 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:39.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 2 == 0 2020-03-11T22:48:39.923 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:39.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:39.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:39.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:39.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:39.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:39.925 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:40.261 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:40.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:40.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:40.262 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:40.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:40.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:48:40.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:48:40.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:48:40.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:40.721 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:40.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:40.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:40.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:40.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:40.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:40.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:40.822 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:40.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:41.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:41.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:41.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:41.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:41.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:41.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:41.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:41.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:41.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:41.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:41.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:41.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:41.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:42.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23063256 2020-03-11T22:48:42.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23063256 '!=' null 2020-03-11T22:48:42.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:42.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:42.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:42.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:42.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:42.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:42.522 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:42.522 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:42.522 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:42.522 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:42.522 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:42.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:42.900 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:42.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:42.901 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:43.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:43.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:43.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:43.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:43.357 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:43.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:43.358 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:43.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:43.358 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:43.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23063256 2020-03-11T22:48:43.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23063256 '!=' null 2020-03-11T22:48:43.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:43.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:43.807 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:43.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:43.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:43.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:43.909 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:43.909 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:43.909 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:43.909 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:43.909 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:44.247 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:44.248 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:44.248 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:44.248 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:44.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:44.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:44.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:44.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:44.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:44.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:44.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:44.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:44.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:45.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=26250126 2020-03-11T22:48:45.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 26250126 '!=' null 2020-03-11T22:48:45.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:45.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:45.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:45.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:45.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:45.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:45.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:45.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:45.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:45.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:45.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:46.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:46.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:46.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:46.005 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:46.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:46.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:46.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:46.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:46.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:46.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:46.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:46.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:46.774 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33197378 2020-03-11T22:48:47.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33197378 '!=' null 2020-03-11T22:48:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:47.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:47.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:47.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:47.338 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:47.338 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:47.338 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:47.338 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:47.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:47.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:47.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:47.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:47.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:47.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:48.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:48.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:48.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:48.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:48.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:48.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:48.129 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:48.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:48.130 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:48.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28829920 2020-03-11T22:48:48.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28829920 '!=' null 2020-03-11T22:48:48.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:48.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:48.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:48.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:48.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:48.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:48.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:48.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:48.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:48.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:48.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:49.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:49.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:49.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:49.436 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:49.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:49.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:49.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:49.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:49.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:49.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:49.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:49.968 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:49.968 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:50.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33198578 2020-03-11T22:48:50.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33198578 '!=' null 2020-03-11T22:48:50.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:50.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:50.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:50.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:50.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:50.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:50.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:50.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:50.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:50.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:50.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:50.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:50.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:50.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:50.965 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:51.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:51.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:51.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:51.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:51.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:51.496 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:51.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:51.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:51.497 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:52.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957046 2020-03-11T22:48:52.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957046 '!=' null 2020-03-11T22:48:52.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:52.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:52.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:52.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:52.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:52.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:52.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:52.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:52.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:52.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:52.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:52.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:52.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:52.682 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:52.682 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:53.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:53.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:53.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:53.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:53.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:53.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:53.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:53.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:53.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:53.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957046 2020-03-11T22:48:53.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957046 '!=' null 2020-03-11T22:48:53.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:53.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:53.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:53.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:53.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:53.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:53.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:53.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:53.852 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:53.853 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:53.853 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:54.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:54.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:54.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:54.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:54.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:54.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:54.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:54.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:54.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:54.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:54.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:54.650 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:54.650 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:55.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34946278 2020-03-11T22:48:55.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34946278 '!=' null 2020-03-11T22:48:55.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:55.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:55.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:55.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:55.443 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:55.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:55.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:55.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:55.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:55.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:55.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:55.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:55.829 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:55.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:55.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:56.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:56.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:56.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:56.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:56.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:56.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:56.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:56.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:56.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:56.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34946067 2020-03-11T22:48:56.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34946067 '!=' null 2020-03-11T22:48:56.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:56.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:56.847 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:56.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:56.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:56.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:56.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:56.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:56.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:56.949 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:56.950 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:57.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:57.331 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:57.331 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:57.331 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:57.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:57.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:57.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:57.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:57.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:57.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:57.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:57.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:57.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:58.341 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27956866 2020-03-11T22:48:58.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27956866 '!=' null 2020-03-11T22:48:58.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:58.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:58.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:58.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:58.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:58.443 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:58.443 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:58.443 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:58.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:58.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:58.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:48:58.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:48:58.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:48:58.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:48:58.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:48:59.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:48:59.236 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:48:59.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:48:59.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:48:59.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:48:59.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:48:59.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:48:59.238 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:48:59.238 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:48:59.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=41935732 2020-03-11T22:48:59.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 41935732 '!=' null 2020-03-11T22:48:59.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:48:59.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:48:59.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:48:59.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:48:59.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:48:59.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:48:59.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:48:59.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:48:59.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:48:59.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:48:59.985 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:00.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:00.396 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:00.396 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:00.397 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:00.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:00.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:00.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:00.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:00.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:00.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:00.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:00.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:00.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:01.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=41935732 2020-03-11T22:49:01.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 41935732 '!=' null 2020-03-11T22:49:01.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:01.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:01.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:01.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:01.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:01.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:01.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:01.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:01.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:01.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:01.697 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:02.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:02.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:02.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:02.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:02.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:02.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:02.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:02.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:02.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:02.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:02.486 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:02.486 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:02.486 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:03.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27957124 2020-03-11T22:49:03.022 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27957124 '!=' null 2020-03-11T22:49:03.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:03.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:03.023 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:03.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:03.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:03.124 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:03.124 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:03.124 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:03.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:03.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:03.125 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:03.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:03.526 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:03.526 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:03.526 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:03.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:03.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:03.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:03.978 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:03.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:03.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:03.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:03.979 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:03.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:04.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=41936122 2020-03-11T22:49:04.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 41936122 '!=' null 2020-03-11T22:49:04.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:04.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:04.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:04.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:04.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:04.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:04.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:04.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:04.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:04.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:04.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:04.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:49:04.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:04.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:04.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:05.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 2 2020-03-11T22:49:05.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:49:05.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:05.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:05.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:05.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:05.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:05.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:05.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:05.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=34072814 2020-03-11T22:49:05.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 34072814 '!=' null 2020-03-11T22:49:05.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:05.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:05.801 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:05.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:05.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:05.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:05.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:05.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:05.902 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:05.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:05.903 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:06.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:49:06.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:06.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:06.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:06.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:49:06.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:49:06.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:49:06.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:49:06.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:06.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:06.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:06.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:06.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:06.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:06.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:06.783 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:06.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:06.784 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:07.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:49:07.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:07.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:07.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:49:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:49:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:07.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:07.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:07.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:07.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:07.566 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:07.566 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:08.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=11457785 2020-03-11T22:49:08.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 11457785 '!=' null 2020-03-11T22:49:08.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:08.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:08.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:08.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:08.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:08.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:08.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:08.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:08.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:08.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:08.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:08.451 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:49:08.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:08.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:08.452 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:08.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 2 2020-03-11T22:49:08.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:49:08.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:49:08.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:49:08.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:49:08.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:49:08.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:49:08.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:49:08.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:49:09.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=11457785 2020-03-11T22:49:09.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 11457785 '!=' null 2020-03-11T22:49:09.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:49:09.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:49:09.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:49:09.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:49:09.466 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:49:09.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:49:09.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:49:09.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:49:09.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:49:09.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:49:09.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:49:09.805 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:49:09.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:49:09.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:49:09.806 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:49:10.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 2 2020-03-11T22:49:10.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:49:10.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:49:10.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:502: TEST_backfill_pool_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:49:10.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:506: TEST_backfill_pool_priority: echo TEST PASSED 2020-03-11T22:49:10.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:509: TEST_backfill_pool_priority: delete_pool test1 2020-03-11T22:49:10.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:49:10.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:49:10.251 INFO:tasks.workunit.client.0.smithi155.stdout:TEST PASSED 2020-03-11T22:49:10.717 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test1' does not exist 2020-03-11T22:49:10.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:510: TEST_backfill_pool_priority: delete_pool test2 2020-03-11T22:49:10.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:49:10.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:49:11.280 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test2' does not exist 2020-03-11T22:49:11.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:511: TEST_backfill_pool_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:49:11.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:11.290 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:11.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:11.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:11.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:11.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:11.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:512: TEST_backfill_pool_priority: return 0 2020-03-11T22:49:11.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:49:11.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:49:11.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:49:11.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:49:11.399 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:11.399 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:11.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:11.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:11.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:11.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:11.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:49:11.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:49:11.403 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:49:11.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:49:11.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:49:11.404 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:49:11.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:11.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:49:11.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:49:11.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:11.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:49:11.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:49:11.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:49:11.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:49:11.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:11.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:49:11.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56110 2020-03-11T22:49:11.420 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:49:11.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:49:11.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:37: run: for func in $funcs 2020-03-11T22:49:11.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:38: run: setup td/osd-backfill-prio 2020-03-11T22:49:11.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-prio 2020-03-11T22:49:11.421 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-prio 2020-03-11T22:49:11.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:49:11.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:49:11.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:49:11.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:49:11.422 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:49:11.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:49:11.422 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:49:11.423 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:49:11.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:49:11.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:49:11.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:49:11.424 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:49:11.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:49:11.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:49:11.425 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:49:11.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:11.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:49:11.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:49:11.426 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:49:11.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:49:11.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:49:11.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:49:11.429 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:49:11.429 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:11.429 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:49:11.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56110 2020-03-11T22:49:11.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:49:11.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:49:11.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-prio 2020-03-11T22:49:11.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:49:11.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:11.431 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:49:11.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.56110 2020-03-11T22:49:11.433 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:49:11.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:49:11.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:49:11.433 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-prio 1' TERM HUP INT 2020-03-11T22:49:11.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:39: run: TEST_backfill_priority td/osd-backfill-prio 2020-03-11T22:49:11.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:46: TEST_backfill_priority: local dir=td/osd-backfill-prio 2020-03-11T22:49:11.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:47: TEST_backfill_priority: local pools=10 2020-03-11T22:49:11.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:48: TEST_backfill_priority: local OSDS=5 2020-03-11T22:49:11.434 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: expr 150 + 1 2020-03-11T22:49:11.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:50: TEST_backfill_priority: local degraded_prio=151 2020-03-11T22:49:11.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:51: TEST_backfill_priority: local max_tries=10 2020-03-11T22:49:11.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:53: TEST_backfill_priority: run_mon td/osd-backfill-prio a 2020-03-11T22:49:11.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-prio 2020-03-11T22:49:11.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:49:11.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:49:11.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:49:11.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-prio/a 2020-03-11T22:49:11.436 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-prio/a --run-dir=td/osd-backfill-prio 2020-03-11T22:49:11.728 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:49:11.728 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:11.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:11.729 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:11.729 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:11.729 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:49:11.729 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:49:11.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-prio/a '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-prio/log --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:49:11.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:49:11.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:49:11.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:49:11.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:49:11.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:49:11.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:49:11.881 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:49:11.881 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:49:11.881 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:49:11.882 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:11.882 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:11.882 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:49:11.882 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-mon.a.asok 2020-03-11T22:49:11.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:49:11.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56110/ceph-mon.a.asok config get fsid 2020-03-11T22:49:11.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:49:11.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:49:11.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:49:11.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:49:11.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:49:11.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:49:11.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:49:11.970 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:49:11.970 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:49:11.971 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:11.971 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:49:11.971 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-mon.a.asok 2020-03-11T22:49:11.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:49:11.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.56110/ceph-mon.a.asok config get mon_host 2020-03-11T22:49:12.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:54: TEST_backfill_priority: run_mgr td/osd-backfill-prio x 2020-03-11T22:49:12.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-prio 2020-03-11T22:49:12.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:49:12.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:49:12.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:49:12.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-prio/x 2020-03-11T22:49:12.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:49:12.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:49:12.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:12.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:12.955 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:12.955 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:12.955 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:49:12.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:49:12.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:49:12.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-prio/x '--log-file=td/osd-backfill-prio/$name.log' '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --run-dir=td/osd-backfill-prio '--pid-file=td/osd-backfill-prio/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:49:12.979 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:55: TEST_backfill_priority: export CEPH_ARGS 2020-03-11T22:49:12.979 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: expr 5 - 1 2020-03-11T22:49:12.980 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: seq 0 4 2020-03-11T22:49:12.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:49:12.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 0 2020-03-11T22:49:12.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:49:12.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:12.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:49:12.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:12.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/0 2020-03-11T22:49:12.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:12.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:12.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:12.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:12.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/0' 2020-03-11T22:49:12.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/0/journal' 2020-03-11T22:49:12.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:12.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:12.986 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:49:12.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:12.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:12.986 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:12.986 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:12.987 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:12.987 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:49:12.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:49:12.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:49:12.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:12.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:12.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:12.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:49:12.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:49:12.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:12.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:12.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:12.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:12.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:12.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/0 2020-03-11T22:49:12.989 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:12.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=77da630d-2760-4396-8b1f-dcac5ed5304a 2020-03-11T22:49:12.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 77da630d-2760-4396-8b1f-dcac5ed5304a' 2020-03-11T22:49:12.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:12.990 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 77da630d-2760-4396-8b1f-dcac5ed5304a 2020-03-11T22:49:13.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDoamleCV+COxAAN33TxbomCB4yjIYO3EBRaQ== 2020-03-11T22:49:13.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDoamleCV+COxAAN33TxbomCB4yjIYO3EBRaQ=="}' 2020-03-11T22:49:13.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 77da630d-2760-4396-8b1f-dcac5ed5304a -i td/osd-backfill-prio/0/new.json 2020-03-11T22:49:13.369 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:49:13.378 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/0/new.json 2020-03-11T22:49:13.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDoamleCV+COxAAN33TxbomCB4yjIYO3EBRaQ== --osd-uuid 77da630d-2760-4396-8b1f-dcac5ed5304a 2020-03-11T22:49:13.404 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:13.402+0000 7fa1d2370ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:13.406 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:13.404+0000 7fa1d2370ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:13.408 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:13.406+0000 7fa1d2370ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:13.409 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:13.406+0000 7fa1d2370ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:49:13.409 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:13.406+0000 7fa1d2370ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:49:13.409 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:13.406+0000 7fa1d2370ec0 -1 bluestore(td/osd-backfill-prio/0/block) _read_bdev_label failed to open td/osd-backfill-prio/0/block: (2) No such file or directory 2020-03-11T22:49:13.454 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:13.451+0000 7fa1d2370ec0 -1 bluestore(td/osd-backfill-prio/0) _read_fsid unparsable uuid 2020-03-11T22:49:16.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/0/keyring 2020-03-11T22:49:16.431 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:16.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:49:16.432 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:16.432 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2020-03-11T22:49:16.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:49:16.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:16.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:16.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:16.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/0 --osd-journal=td/osd-backfill-prio/0/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:16.886 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:49:16.909 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:16.906+0000 7f11adca1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:16.910 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:16.908+0000 7f11adca1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:16.912 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:16.909+0000 7f11adca1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:17.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:49:17.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:17.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:49:17.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:17.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:17.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:17.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:17.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:17.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:49:17.243 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:49:17.427 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:17.424+0000 7f11adca1ec0 -1 Falling back to public interface 2020-03-11T22:49:17.590 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:18.178 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:18.175+0000 7f11adca1ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:49:18.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:18.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:18.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:18.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:18.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:49:18.592 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:49:18.951 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/63054,v1:127.0.0.1:6803/63054] [v2:127.0.0.1:6804/63054,v1:127.0.0.1:6805/63054] exists,up 77da630d-2760-4396-8b1f-dcac5ed5304a 2020-03-11T22:49:18.951 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 7c4fee3a-7281-4b1e-ae35-77a23a1f3380 2020-03-11T22:49:18.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:18.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:18.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:18.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:49:18.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 1 2020-03-11T22:49:18.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:49:18.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:18.952 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:49:18.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:18.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/1 2020-03-11T22:49:18.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:18.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:18.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:18.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:18.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/1' 2020-03-11T22:49:18.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/1/journal' 2020-03-11T22:49:18.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:18.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:18.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:49:18.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:18.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:18.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:18.954 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:18.955 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:18.955 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:49:18.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:49:18.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:49:18.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:18.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:18.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:18.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:49:18.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:49:18.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:18.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:18.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:18.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:18.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:18.956 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/1 2020-03-11T22:49:18.956 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:18.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=7c4fee3a-7281-4b1e-ae35-77a23a1f3380 2020-03-11T22:49:18.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 7c4fee3a-7281-4b1e-ae35-77a23a1f3380' 2020-03-11T22:49:18.957 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:18.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDuamle8Bt7ORAA+OSZHVzSHIwnWRWb3Bx1BA== 2020-03-11T22:49:18.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDuamle8Bt7ORAA+OSZHVzSHIwnWRWb3Bx1BA=="}' 2020-03-11T22:49:18.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 7c4fee3a-7281-4b1e-ae35-77a23a1f3380 -i td/osd-backfill-prio/1/new.json 2020-03-11T22:49:19.465 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:49:19.475 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/1/new.json 2020-03-11T22:49:19.476 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDuamle8Bt7ORAA+OSZHVzSHIwnWRWb3Bx1BA== --osd-uuid 7c4fee3a-7281-4b1e-ae35-77a23a1f3380 2020-03-11T22:49:19.501 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:19.497+0000 7fae9bd23ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:19.530 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:19.527+0000 7fae9bd23ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:19.533 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:19.529+0000 7fae9bd23ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:19.533 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:19.530+0000 7fae9bd23ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:49:19.534 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:19.530+0000 7fae9bd23ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:49:19.534 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:19.530+0000 7fae9bd23ec0 -1 bluestore(td/osd-backfill-prio/1/block) _read_bdev_label failed to open td/osd-backfill-prio/1/block: (2) No such file or directory 2020-03-11T22:49:19.634 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:19.631+0000 7fae9bd23ec0 -1 bluestore(td/osd-backfill-prio/1) _read_fsid unparsable uuid 2020-03-11T22:49:22.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/1/keyring 2020-03-11T22:49:22.728 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:22.729 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2020-03-11T22:49:22.729 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:49:22.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:23.219 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:49:23.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:49:23.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/1 --osd-journal=td/osd-backfill-prio/1/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:23.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:23.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:23.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:23.244 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:23.240+0000 7f51491f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:23.246 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:23.242+0000 7f51491f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:23.247 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:23.244+0000 7f51491f4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:23.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:49:23.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:23.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:49:23.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:23.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:23.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:23.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:23.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:23.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:49:23.566 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:49:23.766 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:23.762+0000 7f51491f4ec0 -1 Falling back to public interface 2020-03-11T22:49:23.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:24.559 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:24.555+0000 7f51491f4ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:49:24.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:24.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:24.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:24.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:24.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:49:24.947 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:49:25.299 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/63417,v1:127.0.0.1:6811/63417] [v2:127.0.0.1:6812/63417,v1:127.0.0.1:6813/63417] exists,up 7c4fee3a-7281-4b1e-ae35-77a23a1f3380 2020-03-11T22:49:25.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:25.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:25.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:25.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:49:25.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 2 2020-03-11T22:49:25.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:49:25.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:25.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:49:25.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:25.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/2 2020-03-11T22:49:25.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:25.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:25.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:25.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:25.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/2' 2020-03-11T22:49:25.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/2/journal' 2020-03-11T22:49:25.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:25.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:25.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:49:25.302 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:25.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:25.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:25.303 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:25.303 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:25.303 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:49:25.303 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:49:25.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:49:25.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:25.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:25.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:25.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:49:25.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:49:25.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:25.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:25.304 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:25.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:25.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:25.305 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/2 2020-03-11T22:49:25.305 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:25.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=0a6c8815-dd00-4ca9-b083-0e410c90a17e 2020-03-11T22:49:25.306 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 0a6c8815-dd00-4ca9-b083-0e410c90a17e' 2020-03-11T22:49:25.306 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:25.306 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 0a6c8815-dd00-4ca9-b083-0e410c90a17e 2020-03-11T22:49:25.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD1amleOgyxEhAAscWqbk+JR4PVHXrYczwo1A== 2020-03-11T22:49:25.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD1amleOgyxEhAAscWqbk+JR4PVHXrYczwo1A=="}' 2020-03-11T22:49:25.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 0a6c8815-dd00-4ca9-b083-0e410c90a17e -i td/osd-backfill-prio/2/new.json 2020-03-11T22:49:25.866 INFO:tasks.workunit.client.0.smithi155.stdout:2 2020-03-11T22:49:25.876 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/2/new.json 2020-03-11T22:49:25.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD1amleOgyxEhAAscWqbk+JR4PVHXrYczwo1A== --osd-uuid 0a6c8815-dd00-4ca9-b083-0e410c90a17e 2020-03-11T22:49:25.902 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:25.898+0000 7fe3fcc19ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:25.903 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:25.900+0000 7fe3fcc19ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:25.905 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:25.902+0000 7fe3fcc19ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:25.906 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:25.902+0000 7fe3fcc19ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:49:25.906 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:25.902+0000 7fe3fcc19ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:49:25.906 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:25.902+0000 7fe3fcc19ec0 -1 bluestore(td/osd-backfill-prio/2/block) _read_bdev_label failed to open td/osd-backfill-prio/2/block: (2) No such file or directory 2020-03-11T22:49:25.974 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:25.971+0000 7fe3fcc19ec0 -1 bluestore(td/osd-backfill-prio/2) _read_fsid unparsable uuid 2020-03-11T22:49:29.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/2/keyring 2020-03-11T22:49:29.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:29.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:49:29.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:29.160 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2020-03-11T22:49:29.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:49:29.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:29.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/2 --osd-journal=td/osd-backfill-prio/2/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:29.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:29.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:29.624 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:49:29.644 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:29.641+0000 7f8db1de2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:29.646 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:29.643+0000 7f8db1de2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:29.648 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:29.644+0000 7f8db1de2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:29.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:49:29.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:29.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:49:29.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:29.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:29.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:29.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:29.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:29.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:49:29.998 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:49:30.165 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:30.161+0000 7f8db1de2ec0 -1 Falling back to public interface 2020-03-11T22:49:30.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:30.907 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:30.904+0000 7f8db1de2ec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:49:31.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:31.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:31.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:31.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:31.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:49:31.371 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:49:31.748 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/63773,v1:127.0.0.1:6819/63773] [v2:127.0.0.1:6820/63773,v1:127.0.0.1:6821/63773] exists,up 0a6c8815-dd00-4ca9-b083-0e410c90a17e 2020-03-11T22:49:31.748 INFO:tasks.workunit.client.0.smithi155.stdout:add osd3 0ffa5804-015e-4580-bd32-d146d07bf2eb 2020-03-11T22:49:31.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:31.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:31.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:31.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:49:31.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 3 2020-03-11T22:49:31.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:49:31.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:31.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:49:31.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:31.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/3 2020-03-11T22:49:31.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:31.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:31.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:31.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:31.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/3' 2020-03-11T22:49:31.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/3/journal' 2020-03-11T22:49:31.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:31.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:31.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:49:31.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:31.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:31.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:31.752 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:31.752 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:31.752 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:49:31.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:49:31.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:49:31.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:31.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:31.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:31.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:49:31.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:49:31.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:31.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:31.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:31.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:31.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:31.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/3 2020-03-11T22:49:31.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:31.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=0ffa5804-015e-4580-bd32-d146d07bf2eb 2020-03-11T22:49:31.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 0ffa5804-015e-4580-bd32-d146d07bf2eb' 2020-03-11T22:49:31.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:31.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD7amlets9gLRAAnHJEEiLcN2w7M24qqJTp6g== 2020-03-11T22:49:31.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD7amlets9gLRAAnHJEEiLcN2w7M24qqJTp6g=="}' 2020-03-11T22:49:31.764 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 0ffa5804-015e-4580-bd32-d146d07bf2eb -i td/osd-backfill-prio/3/new.json 2020-03-11T22:49:32.281 INFO:tasks.workunit.client.0.smithi155.stdout:3 2020-03-11T22:49:32.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/3/new.json 2020-03-11T22:49:32.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD7amlets9gLRAAnHJEEiLcN2w7M24qqJTp6g== --osd-uuid 0ffa5804-015e-4580-bd32-d146d07bf2eb 2020-03-11T22:49:32.316 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:32.313+0000 7f9df319aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:32.340 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:32.337+0000 7f9df319aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:32.342 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:32.338+0000 7f9df319aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:32.342 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:32.339+0000 7f9df319aec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:49:32.343 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:32.339+0000 7f9df319aec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:49:32.343 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:32.339+0000 7f9df319aec0 -1 bluestore(td/osd-backfill-prio/3/block) _read_bdev_label failed to open td/osd-backfill-prio/3/block: (2) No such file or directory 2020-03-11T22:49:32.482 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:32.479+0000 7f9df319aec0 -1 bluestore(td/osd-backfill-prio/3) _read_fsid unparsable uuid 2020-03-11T22:49:35.574 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/3/keyring 2020-03-11T22:49:35.575 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:35.576 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd3 key to auth repository 2020-03-11T22:49:35.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:49:35.576 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:36.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:49:36.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/3 --osd-journal=td/osd-backfill-prio/3/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:36.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:36.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:36.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:36.032 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.3 2020-03-11T22:49:36.055 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:36.051+0000 7f681a0f3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:36.057 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:36.053+0000 7f681a0f3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:36.058 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:36.055+0000 7f681a0f3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:36.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:49:36.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:36.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:49:36.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:36.384 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:36.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:36.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:36.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:36.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:49:36.385 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:49:36.574 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:36.571+0000 7f681a0f3ec0 -1 Falling back to public interface 2020-03-11T22:49:36.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:37.314 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:37.311+0000 7f681a0f3ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:49:37.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:37.731 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:37.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:37.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:37.732 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:49:37.732 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:49:38.092 INFO:tasks.workunit.client.0.smithi155.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/64128,v1:127.0.0.1:6827/64128] [v2:127.0.0.1:6828/64128,v1:127.0.0.1:6829/64128] exists,up 0ffa5804-015e-4580-bd32-d146d07bf2eb 2020-03-11T22:49:38.092 INFO:tasks.workunit.client.0.smithi155.stdout:add osd4 2b71ea3d-7ec1-4c10-9f7e-142f383349af 2020-03-11T22:49:38.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:38.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:38.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:38.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:57: TEST_backfill_priority: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:49:38.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:59: TEST_backfill_priority: run_osd td/osd-backfill-prio 4 2020-03-11T22:49:38.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-prio 2020-03-11T22:49:38.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:49:38.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:49:38.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:49:38.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-prio/4 2020-03-11T22:49:38.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 ' 2020-03-11T22:49:38.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:49:38.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:49:38.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:49:38.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-prio/4' 2020-03-11T22:49:38.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-prio/4/journal' 2020-03-11T22:49:38.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:49:38.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+= 2020-03-11T22:49:38.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-prio' 2020-03-11T22:49:38.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:49:38.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:49:38.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:49:38.096 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:49:38.096 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:49:38.096 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:49:38.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:49:38.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' 2020-03-11T22:49:38.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:49:38.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:49:38.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:49:38.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-prio/$name.log' 2020-03-11T22:49:38.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-prio/$name.pid' 2020-03-11T22:49:38.097 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:49:38.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:49:38.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:38.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:49:38.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:49:38.098 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-prio/4 2020-03-11T22:49:38.098 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:49:38.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=2b71ea3d-7ec1-4c10-9f7e-142f383349af 2020-03-11T22:49:38.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 2b71ea3d-7ec1-4c10-9f7e-142f383349af' 2020-03-11T22:49:38.099 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:49:38.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQACa2leoBdQBhAAUFi9B9U55YKSb1WFNKwwFQ== 2020-03-11T22:49:38.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQACa2leoBdQBhAAUFi9B9U55YKSb1WFNKwwFQ=="}' 2020-03-11T22:49:38.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 2b71ea3d-7ec1-4c10-9f7e-142f383349af -i td/osd-backfill-prio/4/new.json 2020-03-11T22:49:38.628 INFO:tasks.workunit.client.0.smithi155.stdout:4 2020-03-11T22:49:38.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-prio/4/new.json 2020-03-11T22:49:38.639 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQACa2leoBdQBhAAUFi9B9U55YKSb1WFNKwwFQ== --osd-uuid 2b71ea3d-7ec1-4c10-9f7e-142f383349af 2020-03-11T22:49:38.664 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:38.660+0000 7f65e5522ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:38.666 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:38.663+0000 7f65e5522ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:38.668 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:38.664+0000 7f65e5522ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:38.668 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:38.665+0000 7f65e5522ec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:49:38.668 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:38.665+0000 7f65e5522ec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:49:38.669 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:38.665+0000 7f65e5522ec0 -1 bluestore(td/osd-backfill-prio/4/block) _read_bdev_label failed to open td/osd-backfill-prio/4/block: (2) No such file or directory 2020-03-11T22:49:38.714 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:38.711+0000 7f65e5522ec0 -1 bluestore(td/osd-backfill-prio/4) _read_fsid unparsable uuid 2020-03-11T22:49:41.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-prio/4/keyring 2020-03-11T22:49:41.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:49:41.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:49:41.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-prio/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:49:41.834 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd4 key to auth repository 2020-03-11T22:49:42.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:49:42.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=2529ab5a-d843-4f02-aa31-5b63175d7e3d --auth-supported=none --mon-host=127.0.0.1:7114 --osd_max_backfills=1 --debug_reserver=20 --osd_min_pg_log_entries=5 --osd_max_pg_log_entries=10 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-prio/4 --osd-journal=td/osd-backfill-prio/4/journal --chdir= --run-dir=td/osd-backfill-prio '--admin-socket=/tmp/ceph-asok.56110/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-prio/$name.log' '--pid-file=td/osd-backfill-prio/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:49:42.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:49:42.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:49:42.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:49:42.295 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.4 2020-03-11T22:49:42.318 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:42.315+0000 7f092d162ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:42.320 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:42.317+0000 7f092d162ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:42.321 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:42.318+0000 7f092d162ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:49:42.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:49:42.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:49:42.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:49:42.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:49:42.710 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:49:42.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:42.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:49:42.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:42.711 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:49:42.711 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:49:42.839 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:42.835+0000 7f092d162ec0 -1 Falling back to public interface 2020-03-11T22:49:43.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:49:43.589 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:49:43.586+0000 7f092d162ec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:49:44.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:49:44.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:49:44.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:49:44.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:49:44.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:49:44.061 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:49:44.461 INFO:tasks.workunit.client.0.smithi155.stdout:osd.4 up in weight 1 up_from 24 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/64483,v1:127.0.0.1:6835/64483] [v2:127.0.0.1:6836/64483,v1:127.0.0.1:6837/64483] exists,up 2b71ea3d-7ec1-4c10-9f7e-142f383349af 2020-03-11T22:49:44.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:49:44.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:49:44.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:49:44.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: seq 1 10 2020-03-11T22:49:44.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:44.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test1 1 1 2020-03-11T22:49:44.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test1 1 1 2020-03-11T22:49:45.078 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test1' already exists 2020-03-11T22:49:45.085 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:46.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test1 size 2 2020-03-11T22:49:46.964 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 1 size to 2 2020-03-11T22:49:46.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:46.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test2 1 1 2020-03-11T22:49:46.972 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test2 1 1 2020-03-11T22:49:47.470 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test2' already exists 2020-03-11T22:49:47.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:48.479 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test2 size 2 2020-03-11T22:49:49.177 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 2 size to 2 2020-03-11T22:49:49.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:49.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test3 1 1 2020-03-11T22:49:49.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test3 1 1 2020-03-11T22:49:49.794 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test3' already exists 2020-03-11T22:49:49.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:50.803 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:49:51.543 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 3 size to 2 2020-03-11T22:49:51.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:51.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test4 1 1 2020-03-11T22:49:51.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test4 1 1 2020-03-11T22:49:52.159 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test4' already exists 2020-03-11T22:49:52.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:53.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test4 size 2 2020-03-11T22:49:53.898 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 4 size to 2 2020-03-11T22:49:53.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:53.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test5 1 1 2020-03-11T22:49:53.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test5 1 1 2020-03-11T22:49:54.499 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test5' already exists 2020-03-11T22:49:54.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:55.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test5 size 2 2020-03-11T22:49:56.162 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 5 size to 2 2020-03-11T22:49:56.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:56.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test6 1 1 2020-03-11T22:49:56.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test6 1 1 2020-03-11T22:49:56.754 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test6' already exists 2020-03-11T22:49:56.762 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:49:57.763 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test6 size 2 2020-03-11T22:49:58.476 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 6 size to 2 2020-03-11T22:49:58.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:49:58.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test7 1 1 2020-03-11T22:49:58.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test7 1 1 2020-03-11T22:49:59.137 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test7' already exists 2020-03-11T22:49:59.144 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:50:00.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:50:00.925 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 7 size to 2 2020-03-11T22:50:00.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:00.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test8 1 1 2020-03-11T22:50:00.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test8 1 1 2020-03-11T22:50:01.433 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test8' already exists 2020-03-11T22:50:01.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:50:02.442 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test8 size 2 2020-03-11T22:50:03.099 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 8 size to 2 2020-03-11T22:50:03.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:03.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test9 1 1 2020-03-11T22:50:03.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test9 1 1 2020-03-11T22:50:03.767 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test9' already exists 2020-03-11T22:50:03.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:50:04.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test9 size 2 2020-03-11T22:50:05.387 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 9 size to 2 2020-03-11T22:50:05.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:62: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:05.395 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:64: TEST_backfill_priority: create_pool test10 1 1 2020-03-11T22:50:05.396 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test10 1 1 2020-03-11T22:50:06.003 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test10' already exists 2020-03-11T22:50:06.010 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:50:07.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:65: TEST_backfill_priority: ceph osd pool set test10 size 2 2020-03-11T22:50:07.951 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 10 size to 2 2020-03-11T22:50:07.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:67: TEST_backfill_priority: sleep 5 2020-03-11T22:50:12.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:69: TEST_backfill_priority: wait_for_clean 2020-03-11T22:50:12.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:50:12.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:50:12.962 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:50:12.963 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:50:12.963 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:50:12.963 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:50:12.963 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:50:12.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:50:12.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:50:12.964 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:50:13.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:50:13.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:50:13.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:50:13.065 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:13.065 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:13.410 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:13.410 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:50:13.411 INFO:tasks.workunit.client.0.smithi155.stderr:2 2020-03-11T22:50:13.411 INFO:tasks.workunit.client.0.smithi155.stderr:3 2020-03-11T22:50:13.411 INFO:tasks.workunit.client.0.smithi155.stderr:4' 2020-03-11T22:50:13.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:13.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:13.411 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:13.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:50:13.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:50:13.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:13.537 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:13.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705676 2020-03-11T22:50:13.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676' 2020-03-11T22:50:13.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:13.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:13.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542155 2020-03-11T22:50:13.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-60129542155' 2020-03-11T22:50:13.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:13.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:13.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378634 2020-03-11T22:50:13.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-60129542155 3-81604378634' 2020-03-11T22:50:13.917 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:13.918 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:14.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215112 2020-03-11T22:50:14.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-60129542155 3-81604378634 4-103079215112' 2020-03-11T22:50:14.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:14.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:50:14.043 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:14.044 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:14.044 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:50:14.044 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:14.045 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:50:14.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:50:14.046 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:14.047 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:50:14.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:50:14.391 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:14.391 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705676 2020-03-11T22:50:14.392 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:14.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:14.392 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705676 2020-03-11T22:50:14.393 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:14.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705676 2020-03-11T22:50:14.394 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2020-03-11T22:50:14.394 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705676 2020-03-11T22:50:14.394 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:14.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705676 -lt 38654705676 2020-03-11T22:50:14.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:14.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542155 2020-03-11T22:50:14.740 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:14.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:14.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542155 2020-03-11T22:50:14.741 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:14.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542155 2020-03-11T22:50:14.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542155' 2020-03-11T22:50:14.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:14.743 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542155 2020-03-11T22:50:15.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542155 -lt 60129542155 2020-03-11T22:50:15.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:15.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378634 2020-03-11T22:50:15.090 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:15.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:15.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378634 2020-03-11T22:50:15.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:15.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378634 2020-03-11T22:50:15.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378634' 2020-03-11T22:50:15.092 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:15.092 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 81604378634 2020-03-11T22:50:15.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378634 -lt 81604378634 2020-03-11T22:50:15.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:15.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215112 2020-03-11T22:50:15.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:15.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:15.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215112 2020-03-11T22:50:15.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:15.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215112 2020-03-11T22:50:15.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215112' 2020-03-11T22:50:15.549 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.4 seq 103079215112 2020-03-11T22:50:15.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:15.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215112 -lt 103079215112 2020-03-11T22:50:15.896 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:50:15.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:15.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:16.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 10 == 0 2020-03-11T22:50:16.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:16.382 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:16.383 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:16.383 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:16.383 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:16.383 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:16.383 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:16.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=10 2020-03-11T22:50:16.717 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:16.718 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:16.718 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:17.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 10 = 10 2020-03-11T22:50:17.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:50:17.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:50:17.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:71: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:50:17.486 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:50:17.486 INFO:tasks.workunit.client.0.smithi155.stdout:10.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:07.751936+0000 0'0 78:20 [2,1] 2 [2,1] 2 0'0 2020-03-11T22:50:05.699692+0000 0'0 2020-03-11T22:50:05.699692+0000 0 2020-03-11T22:50:17.486 INFO:tasks.workunit.client.0.smithi155.stdout:9.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:05.900076+0000 0'0 78:25 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:50:03.411100+0000 0'0 2020-03-11T22:50:03.411100+0000 0 2020-03-11T22:50:17.487 INFO:tasks.workunit.client.0.smithi155.stdout:8.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:03.628922+0000 0'0 78:27 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:50:01.237371+0000 0'0 2020-03-11T22:50:01.237371+0000 0 2020-03-11T22:50:17.487 INFO:tasks.workunit.client.0.smithi155.stdout:7.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:50:01.506220+0000 0'0 78:34 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:49:58.789140+0000 0'0 2020-03-11T22:49:58.789140+0000 0 2020-03-11T22:50:17.487 INFO:tasks.workunit.client.0.smithi155.stdout:2.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:50.735018+0000 0'0 78:49 [3,1] 3 [3,1] 3 0'0 2020-03-11T22:49:47.267979+0000 0'0 2020-03-11T22:49:47.267979+0000 0 2020-03-11T22:50:17.487 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:46.791990+0000 0'0 78:52 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:49:44.844495+0000 0'0 2020-03-11T22:49:44.844495+0000 0 2020-03-11T22:50:17.487 INFO:tasks.workunit.client.0.smithi155.stdout:3.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:51.700724+0000 0'0 78:44 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:49:49.548052+0000 0'0 2020-03-11T22:49:49.548052+0000 0 2020-03-11T22:50:17.488 INFO:tasks.workunit.client.0.smithi155.stdout:4.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:54.632609+0000 0'0 76:43 [4,0] 4 [4,0] 4 0'0 2020-03-11T22:49:51.882991+0000 0'0 2020-03-11T22:49:51.882991+0000 0 2020-03-11T22:50:17.488 INFO:tasks.workunit.client.0.smithi155.stdout:5.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:56.533308+0000 0'0 78:38 [3,4] 3 [3,4] 3 0'0 2020-03-11T22:49:54.277104+0000 0'0 2020-03-11T22:49:54.277104+0000 0 2020-03-11T22:50:17.488 INFO:tasks.workunit.client.0.smithi155.stdout:6.0 0 0 0 0 0 0 0 0 0 0 active+clean 2020-03-11T22:49:59.401569+0000 0'0 76:34 [4,2] 4 [4,2] 4 0'0 2020-03-11T22:49:56.574882+0000 0'0 2020-03-11T22:49:56.574882+0000 0 2020-03-11T22:50:17.488 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:50:17.488 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:50:17.489 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:50:17.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:75: TEST_backfill_priority: local PG1 2020-03-11T22:50:17.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:76: TEST_backfill_priority: local POOLNUM1 2020-03-11T22:50:17.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:77: TEST_backfill_priority: local pool1 2020-03-11T22:50:17.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:78: TEST_backfill_priority: local chk_osd1_1 2020-03-11T22:50:17.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:79: TEST_backfill_priority: local chk_osd1_2 2020-03-11T22:50:17.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:81: TEST_backfill_priority: local PG2 2020-03-11T22:50:17.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:82: TEST_backfill_priority: local POOLNUM2 2020-03-11T22:50:17.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:83: TEST_backfill_priority: local pool2 2020-03-11T22:50:17.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:84: TEST_backfill_priority: local chk_osd2 2020-03-11T22:50:17.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:86: TEST_backfill_priority: local PG3 2020-03-11T22:50:17.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:87: TEST_backfill_priority: local POOLNUM3 2020-03-11T22:50:17.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:88: TEST_backfill_priority: local pool3 2020-03-11T22:50:17.498 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: seq 1 10 2020-03-11T22:50:17.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:17.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 1.0 --format=json 2020-03-11T22:50:17.499 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:17.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:17.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:50:17.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:17.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:50:17.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z '' ']' 2020-03-11T22:50:17.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:97: TEST_backfill_priority: PG1=1.0 2020-03-11T22:50:17.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:98: TEST_backfill_priority: POOLNUM1=1 2020-03-11T22:50:17.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:99: TEST_backfill_priority: pool1=test1 2020-03-11T22:50:17.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:100: TEST_backfill_priority: chk_osd1_1=1 2020-03-11T22:50:17.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:101: TEST_backfill_priority: chk_osd1_2=0 2020-03-11T22:50:17.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:17.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 2.0 --format=json 2020-03-11T22:50:17.836 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:18.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:18.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:50:18.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:18.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=1 2020-03-11T22:50:18.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:50:18.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 3 -a 0 '!=' 1 ']' 2020-03-11T22:50:18.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n '' -a 1 = 3 -a 0 '!=' 1 -a '' '!=' 1 ']' 2020-03-11T22:50:18.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:18.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 3.0 --format=json 2020-03-11T22:50:18.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:18.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:18.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:50:18.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:18.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:50:18.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:50:18.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z '' -a 1 = 1 -a 0 '!=' 2 ']' 2020-03-11T22:50:18.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:104: TEST_backfill_priority: PG2=3.0 2020-03-11T22:50:18.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:105: TEST_backfill_priority: POOLNUM2=3 2020-03-11T22:50:18.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:106: TEST_backfill_priority: pool2=test3 2020-03-11T22:50:18.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:107: TEST_backfill_priority: chk_osd2=2 2020-03-11T22:50:18.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:18.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 4.0 --format=json 2020-03-11T22:50:18.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:18.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:18.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:50:18.899 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:18.900 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=0 2020-03-11T22:50:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:50:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 0 ']' 2020-03-11T22:50:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 0 -a 2 '!=' 0 ']' 2020-03-11T22:50:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 5.0 --format=json 2020-03-11T22:50:18.901 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:19.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:19.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=3 2020-03-11T22:50:19.237 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:19.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:50:19.238 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:50:19.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 3 -a 0 '!=' 4 ']' 2020-03-11T22:50:19.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 3 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:50:19.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:19.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 6.0 --format=json 2020-03-11T22:50:19.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:19.602 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:19.603 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=4 2020-03-11T22:50:19.603 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:19.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=2 2020-03-11T22:50:19.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:50:19.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 4 -a 0 '!=' 2 ']' 2020-03-11T22:50:19.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 4 -a 0 '!=' 2 -a 2 '!=' 2 ']' 2020-03-11T22:50:19.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:90: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:19.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: ceph pg map 7.0 --format=json 2020-03-11T22:50:19.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:92: TEST_backfill_priority: jq '.acting[]' 2020-03-11T22:50:19.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: head -1 td/osd-backfill-prio/acting 2020-03-11T22:50:19.944 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:93: TEST_backfill_priority: local test_osd1=1 2020-03-11T22:50:19.944 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: tail -1 td/osd-backfill-prio/acting 2020-03-11T22:50:19.945 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:94: TEST_backfill_priority: local test_osd2=4 2020-03-11T22:50:19.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:95: TEST_backfill_priority: '[' -z 1.0 ']' 2020-03-11T22:50:19.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:102: TEST_backfill_priority: '[' -z 3.0 -a 1 = 1 -a 0 '!=' 4 ']' 2020-03-11T22:50:19.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:108: TEST_backfill_priority: '[' -n 3.0 -a 1 = 1 -a 0 '!=' 4 -a 2 '!=' 4 ']' 2020-03-11T22:50:19.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:110: TEST_backfill_priority: PG3=7.0 2020-03-11T22:50:19.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:111: TEST_backfill_priority: POOLNUM3=7 2020-03-11T22:50:19.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:112: TEST_backfill_priority: pool3=test7 2020-03-11T22:50:19.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:113: TEST_backfill_priority: break 2020-03-11T22:50:19.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:116: TEST_backfill_priority: rm -f td/osd-backfill-prio/acting 2020-03-11T22:50:19.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:118: TEST_backfill_priority: '[' test3 = '' -o pool3 = '' ']' 2020-03-11T22:50:19.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: seq 1 10 2020-03-11T22:50:19.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:19.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 1 '!=' 1 -a 1 '!=' 3 -a 1 '!=' 7 ']' 2020-03-11T22:50:19.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:19.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 2 '!=' 1 -a 2 '!=' 3 -a 2 '!=' 7 ']' 2020-03-11T22:50:19.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test2 2020-03-11T22:50:19.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test2 2020-03-11T22:50:19.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test2 test2 --yes-i-really-really-mean-it 2020-03-11T22:50:20.469 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test2' does not exist 2020-03-11T22:50:20.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:20.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 3 '!=' 1 -a 3 '!=' 3 -a 3 '!=' 7 ']' 2020-03-11T22:50:20.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:20.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 4 '!=' 1 -a 4 '!=' 3 -a 4 '!=' 7 ']' 2020-03-11T22:50:20.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test4 2020-03-11T22:50:20.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test4 2020-03-11T22:50:20.478 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test4 test4 --yes-i-really-really-mean-it 2020-03-11T22:50:20.961 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test4' does not exist 2020-03-11T22:50:20.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:20.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 5 '!=' 1 -a 5 '!=' 3 -a 5 '!=' 7 ']' 2020-03-11T22:50:20.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test5 2020-03-11T22:50:20.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test5 2020-03-11T22:50:20.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test5 test5 --yes-i-really-really-mean-it 2020-03-11T22:50:21.529 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test5' does not exist 2020-03-11T22:50:21.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:21.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 6 '!=' 1 -a 6 '!=' 3 -a 6 '!=' 7 ']' 2020-03-11T22:50:21.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test6 2020-03-11T22:50:21.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test6 2020-03-11T22:50:21.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test6 test6 --yes-i-really-really-mean-it 2020-03-11T22:50:22.104 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test6' does not exist 2020-03-11T22:50:22.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:22.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 7 '!=' 1 -a 7 '!=' 3 -a 7 '!=' 7 ']' 2020-03-11T22:50:22.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:22.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 8 '!=' 1 -a 8 '!=' 3 -a 8 '!=' 7 ']' 2020-03-11T22:50:22.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test8 2020-03-11T22:50:22.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test8 2020-03-11T22:50:22.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test8 test8 --yes-i-really-really-mean-it 2020-03-11T22:50:22.615 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test8' does not exist 2020-03-11T22:50:22.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:22.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 9 '!=' 1 -a 9 '!=' 3 -a 9 '!=' 7 ']' 2020-03-11T22:50:22.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test9 2020-03-11T22:50:22.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test9 2020-03-11T22:50:22.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test9 test9 --yes-i-really-really-mean-it 2020-03-11T22:50:23.176 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test9' does not exist 2020-03-11T22:50:23.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:124: TEST_backfill_priority: for p in $(seq 1 $pools) 2020-03-11T22:50:23.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:126: TEST_backfill_priority: '[' 10 '!=' 1 -a 10 '!=' 3 -a 10 '!=' 7 ']' 2020-03-11T22:50:23.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:128: TEST_backfill_priority: delete_pool test10 2020-03-11T22:50:23.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test10 2020-03-11T22:50:23.185 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test10 test10 --yes-i-really-really-mean-it 2020-03-11T22:50:23.700 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test10' does not exist 2020-03-11T22:50:23.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:132: TEST_backfill_priority: ceph osd pool set test3 size 1 2020-03-11T22:50:24.372 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 3 size to 1 2020-03-11T22:50:24.379 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:133: TEST_backfill_priority: ceph osd pool set test7 size 1 2020-03-11T22:50:25.142 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 7 size to 1 2020-03-11T22:50:25.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:134: TEST_backfill_priority: wait_for_clean 2020-03-11T22:50:25.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:50:25.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:50:25.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:50:25.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:50:25.152 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:50:25.152 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:50:25.153 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:50:25.153 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:50:25.153 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:50:25.153 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:50:25.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:50:25.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:50:25.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:50:25.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:50:25.244 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:50:25.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:50:25.632 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:50:25.632 INFO:tasks.workunit.client.0.smithi155.stderr:2 2020-03-11T22:50:25.633 INFO:tasks.workunit.client.0.smithi155.stderr:3 2020-03-11T22:50:25.633 INFO:tasks.workunit.client.0.smithi155.stderr:4' 2020-03-11T22:50:25.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:50:25.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:25.633 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:50:25.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836498 2020-03-11T22:50:25.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498' 2020-03-11T22:50:25.760 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:25.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:50:25.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705680 2020-03-11T22:50:25.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-38654705680' 2020-03-11T22:50:25.890 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:25.890 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:50:26.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542159 2020-03-11T22:50:26.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-38654705680 2-60129542159' 2020-03-11T22:50:26.013 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:26.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:50:26.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378638 2020-03-11T22:50:26.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-38654705680 2-60129542159 3-81604378638' 2020-03-11T22:50:26.139 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:50:26.139 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:50:26.267 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215117 2020-03-11T22:50:26.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836498 1-38654705680 2-60129542159 3-81604378638 4-103079215117' 2020-03-11T22:50:26.268 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:26.268 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836498 2020-03-11T22:50:26.268 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:26.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:50:26.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836498 2020-03-11T22:50:26.269 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:26.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836498 2020-03-11T22:50:26.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836498' 2020-03-11T22:50:26.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:50:26.271 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836498 2020-03-11T22:50:26.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836498 -lt 21474836498 2020-03-11T22:50:26.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:26.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705680 2020-03-11T22:50:26.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:26.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:50:26.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705680 2020-03-11T22:50:26.620 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:26.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705680 2020-03-11T22:50:26.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705680' 2020-03-11T22:50:26.621 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:50:26.621 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705680 2020-03-11T22:50:26.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705681 -lt 38654705680 2020-03-11T22:50:26.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:26.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542159 2020-03-11T22:50:26.966 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:26.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:50:26.967 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542159 2020-03-11T22:50:26.967 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:26.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542159 2020-03-11T22:50:26.969 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542159' 2020-03-11T22:50:26.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:50:26.969 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542159 2020-03-11T22:50:27.402 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542159 -lt 60129542159 2020-03-11T22:50:27.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:27.403 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378638 2020-03-11T22:50:27.403 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:27.404 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:50:27.404 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378638 2020-03-11T22:50:27.404 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:27.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378638 2020-03-11T22:50:27.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378638' 2020-03-11T22:50:27.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:50:27.406 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 81604378638 2020-03-11T22:50:27.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378638 -lt 81604378638 2020-03-11T22:50:27.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:50:27.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215117 2020-03-11T22:50:27.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:50:27.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:50:27.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215117 2020-03-11T22:50:27.751 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:50:27.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215117 2020-03-11T22:50:27.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215117' 2020-03-11T22:50:27.752 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.4 seq 103079215117 2020-03-11T22:50:27.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:28.099 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215116 -lt 103079215117 2020-03-11T22:50:28.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:50:29.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:50:29.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:50:29.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215117 -lt 103079215117 2020-03-11T22:50:29.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:50:29.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:29.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:29.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:50:29.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:50:29.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:50:29.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:50:29.911 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:50:29.911 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:50:29.911 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:50:29.911 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:50:30.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:50:30.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:50:30.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:50:30.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:50:30.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:50:30.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:50:30.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:50:30.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:136: TEST_backfill_priority: dd if=/dev/urandom of=td/osd-backfill-prio/data bs=1M count=10 2020-03-11T22:50:30.751 INFO:tasks.workunit.client.0.smithi155.stderr:10+0 records in 2020-03-11T22:50:30.751 INFO:tasks.workunit.client.0.smithi155.stderr:10+0 records out 2020-03-11T22:50:30.751 INFO:tasks.workunit.client.0.smithi155.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0462737 s, 227 MB/s 2020-03-11T22:50:30.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:137: TEST_backfill_priority: p=1 2020-03-11T22:50:30.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:50:30.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:50:30.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:30.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj1-p1 td/osd-backfill-prio/data 2020-03-11T22:50:31.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:31.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj2-p1 td/osd-backfill-prio/data 2020-03-11T22:50:31.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:31.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj3-p1 td/osd-backfill-prio/data 2020-03-11T22:50:32.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:32.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj4-p1 td/osd-backfill-prio/data 2020-03-11T22:50:32.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:32.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj5-p1 td/osd-backfill-prio/data 2020-03-11T22:50:33.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:33.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj6-p1 td/osd-backfill-prio/data 2020-03-11T22:50:33.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:33.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj7-p1 td/osd-backfill-prio/data 2020-03-11T22:50:34.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:34.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj8-p1 td/osd-backfill-prio/data 2020-03-11T22:50:34.518 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:34.519 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj9-p1 td/osd-backfill-prio/data 2020-03-11T22:50:35.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:35.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj10-p1 td/osd-backfill-prio/data 2020-03-11T22:50:35.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:35.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj11-p1 td/osd-backfill-prio/data 2020-03-11T22:50:35.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:35.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj12-p1 td/osd-backfill-prio/data 2020-03-11T22:50:36.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:36.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj13-p1 td/osd-backfill-prio/data 2020-03-11T22:50:36.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:36.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj14-p1 td/osd-backfill-prio/data 2020-03-11T22:50:37.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:37.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj15-p1 td/osd-backfill-prio/data 2020-03-11T22:50:37.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:37.725 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj16-p1 td/osd-backfill-prio/data 2020-03-11T22:50:38.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:38.244 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj17-p1 td/osd-backfill-prio/data 2020-03-11T22:50:38.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:38.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj18-p1 td/osd-backfill-prio/data 2020-03-11T22:50:39.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:39.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj19-p1 td/osd-backfill-prio/data 2020-03-11T22:50:39.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:39.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj20-p1 td/osd-backfill-prio/data 2020-03-11T22:50:40.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:40.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj21-p1 td/osd-backfill-prio/data 2020-03-11T22:50:40.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:40.536 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj22-p1 td/osd-backfill-prio/data 2020-03-11T22:50:41.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:41.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj23-p1 td/osd-backfill-prio/data 2020-03-11T22:50:41.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:41.512 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj24-p1 td/osd-backfill-prio/data 2020-03-11T22:50:41.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:41.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj25-p1 td/osd-backfill-prio/data 2020-03-11T22:50:42.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:42.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj26-p1 td/osd-backfill-prio/data 2020-03-11T22:50:42.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:42.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj27-p1 td/osd-backfill-prio/data 2020-03-11T22:50:43.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:43.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj28-p1 td/osd-backfill-prio/data 2020-03-11T22:50:43.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:43.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj29-p1 td/osd-backfill-prio/data 2020-03-11T22:50:44.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:44.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj30-p1 td/osd-backfill-prio/data 2020-03-11T22:50:44.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:44.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj31-p1 td/osd-backfill-prio/data 2020-03-11T22:50:45.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:45.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj32-p1 td/osd-backfill-prio/data 2020-03-11T22:50:45.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:45.662 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj33-p1 td/osd-backfill-prio/data 2020-03-11T22:50:46.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:46.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj34-p1 td/osd-backfill-prio/data 2020-03-11T22:50:46.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:46.712 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj35-p1 td/osd-backfill-prio/data 2020-03-11T22:50:47.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:47.196 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj36-p1 td/osd-backfill-prio/data 2020-03-11T22:50:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:47.704 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj37-p1 td/osd-backfill-prio/data 2020-03-11T22:50:48.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:48.220 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj38-p1 td/osd-backfill-prio/data 2020-03-11T22:50:48.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:48.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj39-p1 td/osd-backfill-prio/data 2020-03-11T22:50:49.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:49.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj40-p1 td/osd-backfill-prio/data 2020-03-11T22:50:49.554 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:49.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj41-p1 td/osd-backfill-prio/data 2020-03-11T22:50:50.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:50.005 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj42-p1 td/osd-backfill-prio/data 2020-03-11T22:50:50.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:50.488 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj43-p1 td/osd-backfill-prio/data 2020-03-11T22:50:50.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:50.896 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj44-p1 td/osd-backfill-prio/data 2020-03-11T22:50:51.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:51.380 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj45-p1 td/osd-backfill-prio/data 2020-03-11T22:50:51.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:51.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj46-p1 td/osd-backfill-prio/data 2020-03-11T22:50:52.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:52.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj47-p1 td/osd-backfill-prio/data 2020-03-11T22:50:52.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:52.802 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj48-p1 td/osd-backfill-prio/data 2020-03-11T22:50:53.277 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:53.278 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj49-p1 td/osd-backfill-prio/data 2020-03-11T22:50:53.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:53.730 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test1 put obj50-p1 td/osd-backfill-prio/data 2020-03-11T22:50:54.233 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 1 + 1 2020-03-11T22:50:54.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=2 2020-03-11T22:50:54.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:50:54.235 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:50:54.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:54.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj1-p2 td/osd-backfill-prio/data 2020-03-11T22:50:54.500 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:54.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj2-p2 td/osd-backfill-prio/data 2020-03-11T22:50:54.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:54.783 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj3-p2 td/osd-backfill-prio/data 2020-03-11T22:50:55.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:55.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj4-p2 td/osd-backfill-prio/data 2020-03-11T22:50:55.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:55.284 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj5-p2 td/osd-backfill-prio/data 2020-03-11T22:50:55.567 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:55.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj6-p2 td/osd-backfill-prio/data 2020-03-11T22:50:55.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:55.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj7-p2 td/osd-backfill-prio/data 2020-03-11T22:50:56.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:56.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj8-p2 td/osd-backfill-prio/data 2020-03-11T22:50:56.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:56.276 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj9-p2 td/osd-backfill-prio/data 2020-03-11T22:50:56.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:56.727 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj10-p2 td/osd-backfill-prio/data 2020-03-11T22:50:56.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:56.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj11-p2 td/osd-backfill-prio/data 2020-03-11T22:50:57.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:57.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj12-p2 td/osd-backfill-prio/data 2020-03-11T22:50:57.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:57.527 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj13-p2 td/osd-backfill-prio/data 2020-03-11T22:50:57.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:57.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj14-p2 td/osd-backfill-prio/data 2020-03-11T22:50:58.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:58.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj15-p2 td/osd-backfill-prio/data 2020-03-11T22:50:58.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:58.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj16-p2 td/osd-backfill-prio/data 2020-03-11T22:50:58.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:58.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj17-p2 td/osd-backfill-prio/data 2020-03-11T22:50:58.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:58.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj18-p2 td/osd-backfill-prio/data 2020-03-11T22:50:59.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:59.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj19-p2 td/osd-backfill-prio/data 2020-03-11T22:50:59.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:59.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj20-p2 td/osd-backfill-prio/data 2020-03-11T22:50:59.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:59.678 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj21-p2 td/osd-backfill-prio/data 2020-03-11T22:50:59.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:50:59.936 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj22-p2 td/osd-backfill-prio/data 2020-03-11T22:51:00.230 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:00.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj23-p2 td/osd-backfill-prio/data 2020-03-11T22:51:00.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:00.495 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj24-p2 td/osd-backfill-prio/data 2020-03-11T22:51:00.756 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:00.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj25-p2 td/osd-backfill-prio/data 2020-03-11T22:51:01.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:01.020 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj26-p2 td/osd-backfill-prio/data 2020-03-11T22:51:01.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:01.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj27-p2 td/osd-backfill-prio/data 2020-03-11T22:51:01.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:01.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj28-p2 td/osd-backfill-prio/data 2020-03-11T22:51:01.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:01.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj29-p2 td/osd-backfill-prio/data 2020-03-11T22:51:02.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:02.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj30-p2 td/osd-backfill-prio/data 2020-03-11T22:51:02.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:02.412 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj31-p2 td/osd-backfill-prio/data 2020-03-11T22:51:02.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:02.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj32-p2 td/osd-backfill-prio/data 2020-03-11T22:51:03.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:03.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj33-p2 td/osd-backfill-prio/data 2020-03-11T22:51:03.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:03.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj34-p2 td/osd-backfill-prio/data 2020-03-11T22:51:03.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:03.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj35-p2 td/osd-backfill-prio/data 2020-03-11T22:51:03.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:03.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj36-p2 td/osd-backfill-prio/data 2020-03-11T22:51:04.087 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:04.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj37-p2 td/osd-backfill-prio/data 2020-03-11T22:51:04.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:04.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj38-p2 td/osd-backfill-prio/data 2020-03-11T22:51:04.604 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:04.605 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj39-p2 td/osd-backfill-prio/data 2020-03-11T22:51:04.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:04.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj40-p2 td/osd-backfill-prio/data 2020-03-11T22:51:05.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:05.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj41-p2 td/osd-backfill-prio/data 2020-03-11T22:51:05.460 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:05.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj42-p2 td/osd-backfill-prio/data 2020-03-11T22:51:05.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:05.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj43-p2 td/osd-backfill-prio/data 2020-03-11T22:51:05.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:05.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj44-p2 td/osd-backfill-prio/data 2020-03-11T22:51:06.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:06.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj45-p2 td/osd-backfill-prio/data 2020-03-11T22:51:06.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:06.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj46-p2 td/osd-backfill-prio/data 2020-03-11T22:51:06.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:06.781 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj47-p2 td/osd-backfill-prio/data 2020-03-11T22:51:07.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:07.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj48-p2 td/osd-backfill-prio/data 2020-03-11T22:51:07.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:07.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj49-p2 td/osd-backfill-prio/data 2020-03-11T22:51:07.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:07.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test3 put obj50-p2 td/osd-backfill-prio/data 2020-03-11T22:51:07.823 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 2 + 1 2020-03-11T22:51:07.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=3 2020-03-11T22:51:07.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:138: TEST_backfill_priority: for pname in $pool1 $pool2 $pool3 2020-03-11T22:51:07.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: seq 1 50 2020-03-11T22:51:07.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:07.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj1-p3 td/osd-backfill-prio/data 2020-03-11T22:51:08.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:08.064 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj2-p3 td/osd-backfill-prio/data 2020-03-11T22:51:08.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:08.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj3-p3 td/osd-backfill-prio/data 2020-03-11T22:51:08.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:08.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj4-p3 td/osd-backfill-prio/data 2020-03-11T22:51:08.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:08.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj5-p3 td/osd-backfill-prio/data 2020-03-11T22:51:09.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:09.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj6-p3 td/osd-backfill-prio/data 2020-03-11T22:51:09.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:09.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj7-p3 td/osd-backfill-prio/data 2020-03-11T22:51:09.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:09.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj8-p3 td/osd-backfill-prio/data 2020-03-11T22:51:09.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:09.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj9-p3 td/osd-backfill-prio/data 2020-03-11T22:51:10.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:10.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj10-p3 td/osd-backfill-prio/data 2020-03-11T22:51:10.582 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:10.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj11-p3 td/osd-backfill-prio/data 2020-03-11T22:51:10.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:10.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj12-p3 td/osd-backfill-prio/data 2020-03-11T22:51:11.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:11.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj13-p3 td/osd-backfill-prio/data 2020-03-11T22:51:11.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:11.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj14-p3 td/osd-backfill-prio/data 2020-03-11T22:51:11.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:11.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj15-p3 td/osd-backfill-prio/data 2020-03-11T22:51:11.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:11.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj16-p3 td/osd-backfill-prio/data 2020-03-11T22:51:12.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:12.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj17-p3 td/osd-backfill-prio/data 2020-03-11T22:51:12.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:12.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj18-p3 td/osd-backfill-prio/data 2020-03-11T22:51:12.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:12.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj19-p3 td/osd-backfill-prio/data 2020-03-11T22:51:12.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:12.959 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj20-p3 td/osd-backfill-prio/data 2020-03-11T22:51:13.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:13.302 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj21-p3 td/osd-backfill-prio/data 2020-03-11T22:51:13.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:13.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj22-p3 td/osd-backfill-prio/data 2020-03-11T22:51:13.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:13.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj23-p3 td/osd-backfill-prio/data 2020-03-11T22:51:14.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:14.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj24-p3 td/osd-backfill-prio/data 2020-03-11T22:51:14.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:14.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj25-p3 td/osd-backfill-prio/data 2020-03-11T22:51:14.717 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:14.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj26-p3 td/osd-backfill-prio/data 2020-03-11T22:51:15.007 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:15.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj27-p3 td/osd-backfill-prio/data 2020-03-11T22:51:15.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:15.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj28-p3 td/osd-backfill-prio/data 2020-03-11T22:51:15.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:15.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj29-p3 td/osd-backfill-prio/data 2020-03-11T22:51:15.826 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:15.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj30-p3 td/osd-backfill-prio/data 2020-03-11T22:51:16.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:16.121 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj31-p3 td/osd-backfill-prio/data 2020-03-11T22:51:16.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:16.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj32-p3 td/osd-backfill-prio/data 2020-03-11T22:51:16.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:16.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj33-p3 td/osd-backfill-prio/data 2020-03-11T22:51:16.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:16.877 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj34-p3 td/osd-backfill-prio/data 2020-03-11T22:51:17.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:17.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj35-p3 td/osd-backfill-prio/data 2020-03-11T22:51:17.510 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:17.511 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj36-p3 td/osd-backfill-prio/data 2020-03-11T22:51:17.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:17.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj37-p3 td/osd-backfill-prio/data 2020-03-11T22:51:18.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:18.103 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj38-p3 td/osd-backfill-prio/data 2020-03-11T22:51:18.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:18.385 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj39-p3 td/osd-backfill-prio/data 2020-03-11T22:51:18.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:18.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj40-p3 td/osd-backfill-prio/data 2020-03-11T22:51:18.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:18.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj41-p3 td/osd-backfill-prio/data 2020-03-11T22:51:19.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:19.161 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj42-p3 td/osd-backfill-prio/data 2020-03-11T22:51:19.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:19.403 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj43-p3 td/osd-backfill-prio/data 2020-03-11T22:51:19.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:19.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj44-p3 td/osd-backfill-prio/data 2020-03-11T22:51:19.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:19.937 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj45-p3 td/osd-backfill-prio/data 2020-03-11T22:51:20.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:20.219 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj46-p3 td/osd-backfill-prio/data 2020-03-11T22:51:20.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:20.454 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj47-p3 td/osd-backfill-prio/data 2020-03-11T22:51:20.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:20.741 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj48-p3 td/osd-backfill-prio/data 2020-03-11T22:51:20.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:20.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj49-p3 td/osd-backfill-prio/data 2020-03-11T22:51:21.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:140: TEST_backfill_priority: for i in $(seq 1 $objects) 2020-03-11T22:51:21.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:142: TEST_backfill_priority: rados -p test7 put obj50-p3 td/osd-backfill-prio/data 2020-03-11T22:51:21.546 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: expr 3 + 1 2020-03-11T22:51:21.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:144: TEST_backfill_priority: p=4 2020-03-11T22:51:21.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: get_not_primary test1 obj1-p1 2020-03-11T22:51:21.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1166: get_not_primary: local poolname=test1 2020-03-11T22:51:21.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1167: get_not_primary: local objectname=obj1-p1 2020-03-11T22:51:21.548 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: get_primary test1 obj1-p1 2020-03-11T22:51:21.548 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: get_primary: local poolname=test1 2020-03-11T22:51:21.548 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: get_primary: local objectname=obj1-p1 2020-03-11T22:51:21.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: get_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:51:21.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1137: get_primary: jq .acting_primary 2020-03-11T22:51:21.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1169: get_not_primary: local primary=1 2020-03-11T22:51:21.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1170: get_not_primary: ceph --format json osd map test1 obj1-p1 2020-03-11T22:51:21.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1171: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2020-03-11T22:51:22.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:147: TEST_backfill_priority: local otherosd=0 2020-03-11T22:51:22.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:149: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:51:22.592 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:51:22.593 INFO:tasks.workunit.client.0.smithi155.stdout:7.0 48 0 0 0 0 497025024 0 0 42 42 active+clean 2020-03-11T22:50:25.586500+0000 91'142 91:191 [1] 1 [1] 1 0'0 2020-03-11T22:49:58.789140+0000 0'0 2020-03-11T22:49:58.789140+0000 0 2020-03-11T22:51:22.593 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:49:46.791990+0000 91'150 91:210 [1,0] 1 [1,0] 1 0'0 2020-03-11T22:49:44.844495+0000 0'0 2020-03-11T22:49:44.844495+0000 0 2020-03-11T22:51:22.593 INFO:tasks.workunit.client.0.smithi155.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:50:24.442902+0000 91'150 91:209 [1] 1 [1] 1 0'0 2020-03-11T22:49:49.548052+0000 0'0 2020-03-11T22:49:49.548052+0000 0 2020-03-11T22:51:22.593 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:51:22.593 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:51:22.594 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:51:22.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:150: TEST_backfill_priority: ERRORS=0 2020-03-11T22:51:22.602 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:152: TEST_backfill_priority: ceph osd set nobackfill 2020-03-11T22:51:23.237 INFO:tasks.workunit.client.0.smithi155.stderr:nobackfill is set 2020-03-11T22:51:23.245 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:153: TEST_backfill_priority: ceph osd set noout 2020-03-11T22:51:24.047 INFO:tasks.workunit.client.0.smithi155.stderr:noout is set 2020-03-11T22:51:24.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:157: TEST_backfill_priority: ceph osd pool set test7 size 2 2020-03-11T22:51:24.749 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 7 size to 2 2020-03-11T22:51:24.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:158: TEST_backfill_priority: sleep 2 2020-03-11T22:51:26.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:26.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:26.760 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:26.760 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:26.760 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:26.760 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:51:26.761 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-osd.1.asok 2020-03-11T22:51:26.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:26.761 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:160: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:26.858 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:51:26.858 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:51:26.859 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:26.859 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:26.859 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:51:26.859 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:51:26.859 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:26.859 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:51:26.859 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:51:26.860 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:26.860 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:26.860 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:26.860 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:51:26.860 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:51:26.860 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:26.860 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:26.860 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:51:26.861 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:51:26.861 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:26.861 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:51:26.866 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: seq 1 10 2020-03-11T22:51:26.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:163: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:51:26.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:51:26.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:165: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:51:27.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:166: TEST_backfill_priority: break 2020-03-11T22:51:27.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:174: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:51:27.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:27.207 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:27.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:27.561 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:51:27.561 INFO:tasks.workunit.client.0.smithi155.stderr:2 2020-03-11T22:51:27.561 INFO:tasks.workunit.client.0.smithi155.stderr:3 2020-03-11T22:51:27.562 INFO:tasks.workunit.client.0.smithi155.stderr:4' 2020-03-11T22:51:27.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:27.562 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:27.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:27.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836512 2020-03-11T22:51:27.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512' 2020-03-11T22:51:27.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:27.688 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:27.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705695 2020-03-11T22:51:27.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512 1-38654705695' 2020-03-11T22:51:27.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:27.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:27.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542174 2020-03-11T22:51:27.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512 1-38654705695 2-60129542174' 2020-03-11T22:51:27.939 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:27.939 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:28.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378652 2020-03-11T22:51:28.066 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512 1-38654705695 2-60129542174 3-81604378652' 2020-03-11T22:51:28.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:28.067 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:28.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215131 2020-03-11T22:51:28.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836512 1-38654705695 2-60129542174 3-81604378652 4-103079215131' 2020-03-11T22:51:28.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:28.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836512 2020-03-11T22:51:28.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:28.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:28.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836512 2020-03-11T22:51:28.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:28.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836512 2020-03-11T22:51:28.195 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836512' 2020-03-11T22:51:28.195 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:28.195 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836512 2020-03-11T22:51:28.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836512 -lt 21474836512 2020-03-11T22:51:28.540 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:28.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705695 2020-03-11T22:51:28.541 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:28.541 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:28.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705695 2020-03-11T22:51:28.542 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:28.542 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705695 2020-03-11T22:51:28.543 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705695' 2020-03-11T22:51:28.543 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705695 2020-03-11T22:51:28.543 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:28.906 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705695 -lt 38654705695 2020-03-11T22:51:28.907 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:28.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542174 2020-03-11T22:51:28.907 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:28.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:51:28.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542174 2020-03-11T22:51:28.908 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:28.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542174 2020-03-11T22:51:28.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542174' 2020-03-11T22:51:28.909 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:51:28.910 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542174 2020-03-11T22:51:29.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542174 -lt 60129542174 2020-03-11T22:51:29.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:29.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378652 2020-03-11T22:51:29.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:29.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:51:29.282 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378652 2020-03-11T22:51:29.282 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:29.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378652 2020-03-11T22:51:29.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378652' 2020-03-11T22:51:29.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:51:29.284 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 81604378652 2020-03-11T22:51:29.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378652 -lt 81604378652 2020-03-11T22:51:29.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:29.673 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215131 2020-03-11T22:51:29.673 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:29.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:51:29.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215131 2020-03-11T22:51:29.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:29.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215131 2020-03-11T22:51:29.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215131' 2020-03-11T22:51:29.675 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.4 seq 103079215131 2020-03-11T22:51:29.676 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:30.054 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215131 -lt 103079215131 2020-03-11T22:51:30.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:30.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:30.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:30.056 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:30.056 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:30.056 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:51:30.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-osd.1.asok 2020-03-11T22:51:30.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:30.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:175: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:30.144 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:51:30.144 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:51:30.144 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:30.144 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:30.145 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:51:30.145 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:51:30.145 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:30.145 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:51:30.145 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:51:30.145 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:30.145 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:30.146 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:30.146 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:51:30.146 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:51:30.146 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:30.146 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:30.146 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:51:30.146 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:51:30.146 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:30.147 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:51:30.151 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:177: TEST_backfill_priority: ceph osd out osd.0 2020-03-11T22:51:30.768 INFO:tasks.workunit.client.0.smithi155.stderr:osd.0 is already out. 2020-03-11T22:51:30.775 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:178: TEST_backfill_priority: sleep 2 2020-03-11T22:51:32.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:179: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:51:32.777 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:32.777 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:33.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:33.135 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:51:33.136 INFO:tasks.workunit.client.0.smithi155.stderr:2 2020-03-11T22:51:33.136 INFO:tasks.workunit.client.0.smithi155.stderr:3 2020-03-11T22:51:33.136 INFO:tasks.workunit.client.0.smithi155.stderr:4' 2020-03-11T22:51:33.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:33.136 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:33.136 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:33.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836515 2020-03-11T22:51:33.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515' 2020-03-11T22:51:33.263 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:33.263 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:33.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705698 2020-03-11T22:51:33.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-38654705698' 2020-03-11T22:51:33.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:33.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:33.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542177 2020-03-11T22:51:33.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-38654705698 2-60129542177' 2020-03-11T22:51:33.517 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:33.517 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:33.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378655 2020-03-11T22:51:33.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-38654705698 2-60129542177 3-81604378655' 2020-03-11T22:51:33.642 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:33.642 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:33.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215134 2020-03-11T22:51:33.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836515 1-38654705698 2-60129542177 3-81604378655 4-103079215134' 2020-03-11T22:51:33.769 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:33.769 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836515 2020-03-11T22:51:33.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:33.770 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:33.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836515 2020-03-11T22:51:33.770 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:33.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836515 2020-03-11T22:51:33.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836515' 2020-03-11T22:51:33.771 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:33.772 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836515 2020-03-11T22:51:34.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836515 2020-03-11T22:51:34.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:51:35.128 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:51:35.129 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:35.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836515 -lt 21474836515 2020-03-11T22:51:35.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:35.485 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705698 2020-03-11T22:51:35.485 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:35.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:35.486 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705698 2020-03-11T22:51:35.486 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:35.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705698 2020-03-11T22:51:35.487 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705698' 2020-03-11T22:51:35.487 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705698 2020-03-11T22:51:35.488 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:35.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705698 -lt 38654705698 2020-03-11T22:51:35.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:35.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542177 2020-03-11T22:51:35.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:35.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:51:35.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542177 2020-03-11T22:51:35.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:35.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542177 2020-03-11T22:51:35.835 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542177' 2020-03-11T22:51:35.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:51:35.835 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542177 2020-03-11T22:51:36.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542177 -lt 60129542177 2020-03-11T22:51:36.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:36.182 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378655 2020-03-11T22:51:36.182 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:36.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:51:36.183 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378655 2020-03-11T22:51:36.183 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:36.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378655 2020-03-11T22:51:36.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378655' 2020-03-11T22:51:36.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:51:36.184 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 81604378655 2020-03-11T22:51:36.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378656 -lt 81604378655 2020-03-11T22:51:36.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:36.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215134 2020-03-11T22:51:36.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:36.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:51:36.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215134 2020-03-11T22:51:36.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:36.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215134 2020-03-11T22:51:36.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215134' 2020-03-11T22:51:36.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:36.550 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.4 seq 103079215134 2020-03-11T22:51:36.897 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215135 -lt 103079215134 2020-03-11T22:51:36.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:36.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:36.897 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:36.897 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:36.898 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:36.898 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:51:36.898 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-osd.1.asok 2020-03-11T22:51:36.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:36.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:180: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:36.985 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:51:36.985 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:51:36.986 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:36.986 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:36.986 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:51:36.986 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:36.986 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:51:36.986 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:51:36.986 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:36.986 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:51:36.987 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:51:36.987 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:36.987 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:36.987 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:36.987 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:36.987 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:51:36.987 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:51:36.987 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:36.988 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:51:36.988 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:51:36.988 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:36.988 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:36.988 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:36.988 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:51:36.988 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:51:36.989 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:36.989 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:36.989 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:51:36.989 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:51:36.989 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:36.989 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:51:36.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:181: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:51:37.317 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:51:37.317 INFO:tasks.workunit.client.0.smithi155.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:51:27.102035+0000 91'150 101:216 [1,4] 1 [1] 1 0'0 2020-03-11T22:49:58.789140+0000 0'0 2020-03-11T22:49:58.789140+0000 0 2020-03-11T22:51:37.317 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:51:33.424147+0000 91'150 102:225 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:49:44.844495+0000 0'0 2020-03-11T22:49:44.844495+0000 0 2020-03-11T22:51:37.317 INFO:tasks.workunit.client.0.smithi155.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:50:24.442902+0000 91'150 101:216 [1] 1 [1] 1 0'0 2020-03-11T22:49:49.548052+0000 0'0 2020-03-11T22:49:49.548052+0000 0 2020-03-11T22:51:37.317 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:51:37.317 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:51:37.318 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:51:37.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:183: TEST_backfill_priority: ceph osd pool set test3 size 2 2020-03-11T22:51:37.982 INFO:tasks.workunit.client.0.smithi155.stderr:set pool 3 size to 2 2020-03-11T22:51:37.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:184: TEST_backfill_priority: sleep 2 2020-03-11T22:51:39.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:185: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:51:39.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:39.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:40.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:40.407 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:51:40.407 INFO:tasks.workunit.client.0.smithi155.stderr:2 2020-03-11T22:51:40.407 INFO:tasks.workunit.client.0.smithi155.stderr:3 2020-03-11T22:51:40.407 INFO:tasks.workunit.client.0.smithi155.stderr:4' 2020-03-11T22:51:40.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:40.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:40.408 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:40.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836519 2020-03-11T22:51:40.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519' 2020-03-11T22:51:40.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:40.534 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:40.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705701 2020-03-11T22:51:40.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519 1-38654705701' 2020-03-11T22:51:40.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:40.660 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:40.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542180 2020-03-11T22:51:40.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519 1-38654705701 2-60129542180' 2020-03-11T22:51:40.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:40.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:40.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378659 2020-03-11T22:51:40.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519 1-38654705701 2-60129542180 3-81604378659' 2020-03-11T22:51:40.912 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:40.912 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:41.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215138 2020-03-11T22:51:41.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836519 1-38654705701 2-60129542180 3-81604378659 4-103079215138' 2020-03-11T22:51:41.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:41.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836519 2020-03-11T22:51:41.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:41.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:41.040 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836519 2020-03-11T22:51:41.040 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:41.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836519 2020-03-11T22:51:41.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836519' 2020-03-11T22:51:41.042 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:41.042 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836519 2020-03-11T22:51:41.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836517 -lt 21474836519 2020-03-11T22:51:41.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:51:42.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:51:42.387 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:42.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836519 -lt 21474836519 2020-03-11T22:51:42.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:42.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705701 2020-03-11T22:51:42.743 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:42.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:42.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705701 2020-03-11T22:51:42.744 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:42.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705701 2020-03-11T22:51:42.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705701' 2020-03-11T22:51:42.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:42.746 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705701 2020-03-11T22:51:43.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705702 -lt 38654705701 2020-03-11T22:51:43.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:43.113 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542180 2020-03-11T22:51:43.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:43.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:51:43.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542180 2020-03-11T22:51:43.115 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:43.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542180 2020-03-11T22:51:43.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542180' 2020-03-11T22:51:43.116 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:51:43.116 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542180 2020-03-11T22:51:43.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542181 -lt 60129542180 2020-03-11T22:51:43.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:43.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378659 2020-03-11T22:51:43.462 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:43.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:51:43.463 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378659 2020-03-11T22:51:43.463 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:43.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378659 2020-03-11T22:51:43.464 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378659' 2020-03-11T22:51:43.465 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:51:43.465 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 81604378659 2020-03-11T22:51:43.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378659 -lt 81604378659 2020-03-11T22:51:43.832 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:43.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215138 2020-03-11T22:51:43.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:43.833 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:51:43.833 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215138 2020-03-11T22:51:43.834 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:43.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215138 2020-03-11T22:51:43.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215138' 2020-03-11T22:51:43.835 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.4 seq 103079215138 2020-03-11T22:51:43.835 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:44.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215138 -lt 103079215138 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:51:44.183 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-osd.1.asok 2020-03-11T22:51:44.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:44.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:186: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:44.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:187: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:44.280 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:51:44.280 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:51:44.280 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:44.280 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:44.280 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:51:44.281 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:44.281 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:51:44.281 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:51:44.281 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:44.281 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:51:44.281 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:51:44.281 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:44.281 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:44.282 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:44.282 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:51:44.282 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:44.282 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:51:44.282 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:51:44.282 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:44.282 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:51:44.282 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:51:44.283 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:44.283 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:44.283 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:44.283 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:44.283 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:51:44.283 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:51:44.283 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:44.283 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:51:44.283 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:51:44.284 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:44.284 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:44.284 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:44.284 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:51:44.284 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:51:44.284 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:44.284 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:44.284 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:51:44.285 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:51:44.285 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:44.285 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:51:44.285 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:188: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:51:44.627 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:51:44.627 INFO:tasks.workunit.client.0.smithi155.stdout:7.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfilling+forced_backfill 2020-03-11T22:51:27.102035+0000 91'150 106:220 [1,4] 1 [1] 1 0'0 2020-03-11T22:49:58.789140+0000 0'0 2020-03-11T22:49:58.789140+0000 0 2020-03-11T22:51:44.627 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 50 0 0 50 0 524288000 0 0 50 50 active+remapped+backfill_wait 2020-03-11T22:51:33.424147+0000 91'150 106:229 [1,2] 1 [1,0] 1 0'0 2020-03-11T22:49:44.844495+0000 0'0 2020-03-11T22:49:44.844495+0000 0 2020-03-11T22:51:44.627 INFO:tasks.workunit.client.0.smithi155.stdout:3.0 50 0 50 0 0 524288000 0 0 50 50 active+undersized+degraded+remapped+backfill_wait 2020-03-11T22:51:39.574177+0000 91'150 107:227 [1,2] 1 [1] 1 0'0 2020-03-11T22:49:49.548052+0000 0'0 2020-03-11T22:49:49.548052+0000 0 2020-03-11T22:51:44.628 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:51:44.628 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:51:44.629 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:51:44.637 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:44.638 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "1.0")).prio' 2020-03-11T22:51:44.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:190: TEST_backfill_priority: PRIO=110 2020-03-11T22:51:44.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:191: TEST_backfill_priority: '[' 110 '!=' 110 ']' 2020-03-11T22:51:44.640 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:44.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:51:44.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:51:44.643 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:198: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:51:44.643 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:199: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:51:44.644 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:44.644 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:51:44.645 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:204: TEST_backfill_priority: PRIO=254 2020-03-11T22:51:44.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:205: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:51:44.646 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: seq 1 10 2020-03-11T22:51:44.646 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:213: TEST_backfill_priority: for i in $(seq 1 $max_tries) 2020-03-11T22:51:44.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: ceph pg force-backfill 3.0 2020-03-11T22:51:44.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:215: TEST_backfill_priority: grep -q 'doesn'\''t require backfilling' 2020-03-11T22:51:44.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:216: TEST_backfill_priority: break 2020-03-11T22:51:44.985 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:224: TEST_backfill_priority: sleep 2 2020-03-11T22:51:46.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:46.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:46.987 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:46.987 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:46.987 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:46.988 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:51:46.988 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-osd.1.asok 2020-03-11T22:51:46.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:46.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:225: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:47.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:226: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:47.094 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:51:47.094 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:51:47.094 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:47.094 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:47.095 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:51:47.095 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:47.095 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:51:47.095 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:51:47.095 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:47.095 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:51:47.095 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:51:47.096 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:47.096 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:47.096 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:47.096 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:51:47.096 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:47.096 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 254, 2020-03-11T22:51:47.096 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:51:47.096 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:47.097 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:51:47.097 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:51:47.097 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:47.097 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:47.097 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:47.097 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:47.097 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:51:47.098 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:51:47.098 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:47.098 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:51:47.098 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:51:47.098 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:47.098 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:47.098 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:47.099 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:51:47.099 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:51:47.099 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:47.099 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:47.099 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:51:47.099 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:51:47.099 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:47.100 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:51:47.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:47.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:51:47.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:227: TEST_backfill_priority: PRIO=254 2020-03-11T22:51:47.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:228: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:51:47.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:233: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:51:47.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:47.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:47.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:47.446 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:51:47.446 INFO:tasks.workunit.client.0.smithi155.stderr:2 2020-03-11T22:51:47.446 INFO:tasks.workunit.client.0.smithi155.stderr:3 2020-03-11T22:51:47.446 INFO:tasks.workunit.client.0.smithi155.stderr:4' 2020-03-11T22:51:47.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:47.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:47.446 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:47.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836522 2020-03-11T22:51:47.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522' 2020-03-11T22:51:47.571 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:47.572 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:47.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705705 2020-03-11T22:51:47.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522 1-38654705705' 2020-03-11T22:51:47.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:47.698 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:47.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542184 2020-03-11T22:51:47.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522 1-38654705705 2-60129542184' 2020-03-11T22:51:47.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:47.825 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:47.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378662 2020-03-11T22:51:47.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522 1-38654705705 2-60129542184 3-81604378662' 2020-03-11T22:51:47.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:47.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:48.077 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215141 2020-03-11T22:51:48.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836522 1-38654705705 2-60129542184 3-81604378662 4-103079215141' 2020-03-11T22:51:48.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:48.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836522 2020-03-11T22:51:48.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:48.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:48.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836522 2020-03-11T22:51:48.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:48.080 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836522 2020-03-11T22:51:48.081 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836522' 2020-03-11T22:51:48.081 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:48.081 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836522 2020-03-11T22:51:48.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836522 -lt 21474836522 2020-03-11T22:51:48.455 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:48.456 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705705 2020-03-11T22:51:48.456 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:48.456 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:51:48.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705705 2020-03-11T22:51:48.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:48.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705705 2020-03-11T22:51:48.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705705' 2020-03-11T22:51:48.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:51:48.458 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705705 2020-03-11T22:51:48.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705705 -lt 38654705705 2020-03-11T22:51:48.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:48.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542184 2020-03-11T22:51:48.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:48.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:51:48.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542184 2020-03-11T22:51:48.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:48.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542184 2020-03-11T22:51:48.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542184' 2020-03-11T22:51:48.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:51:48.811 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542184 2020-03-11T22:51:49.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542184 -lt 60129542184 2020-03-11T22:51:49.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:49.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378662 2020-03-11T22:51:49.175 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:49.175 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:51:49.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378662 2020-03-11T22:51:49.176 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:49.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378662 2020-03-11T22:51:49.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378662' 2020-03-11T22:51:49.177 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:51:49.177 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 81604378662 2020-03-11T22:51:49.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378662 -lt 81604378662 2020-03-11T22:51:49.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:49.525 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215141 2020-03-11T22:51:49.526 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:49.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:51:49.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215141 2020-03-11T22:51:49.527 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:49.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215141 2020-03-11T22:51:49.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215141' 2020-03-11T22:51:49.528 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:51:49.528 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.4 seq 103079215141 2020-03-11T22:51:49.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215141 -lt 103079215141 2020-03-11T22:51:49.875 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:236: TEST_backfill_priority: ceph pg cancel-force-backfill 7.0 2020-03-11T22:51:50.204 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg(s) [7.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:51:50.204 INFO:tasks.workunit.client.0.smithi155.stderr: 2020-03-11T22:51:50.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:237: TEST_backfill_priority: sleep 2 2020-03-11T22:51:52.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:52.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:52.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:52.215 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:52.215 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:52.215 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:51:52.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-osd.1.asok 2020-03-11T22:51:52.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:52.216 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:238: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:52.320 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:239: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:52.321 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:51:52.321 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:51:52.321 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:52.321 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:52.321 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:51:52.322 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:52.322 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:51:52.322 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:51:52.322 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:52.322 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:51:52.322 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:51:52.322 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:52.322 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:52.323 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:52.323 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:51:52.323 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:52.323 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:51:52.323 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:51:52.323 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:52.323 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:51:52.324 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:51:52.324 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:52.324 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:52.324 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:52.324 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:52.324 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:51:52.324 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:51:52.324 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:52.325 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:51:52.325 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:51:52.325 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:52.325 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:52.325 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:52.325 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:51:52.325 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:51:52.325 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:52.326 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:52.326 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:51:52.326 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:51:52.326 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:52.326 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:51:52.327 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:52.327 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "7.0")).prio' 2020-03-11T22:51:52.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:240: TEST_backfill_priority: PRIO=151 2020-03-11T22:51:52.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:241: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:51:52.328 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:52.328 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:51:52.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: eval 'ITEM="3.0"' 2020-03-11T22:51:52.328 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:247: TEST_backfill_priority: ITEM=3.0 2020-03-11T22:51:52.328 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:248: TEST_backfill_priority: '[' 3.0 '!=' 3.0 ']' 2020-03-11T22:51:52.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:51:52.329 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:51:52.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:253: TEST_backfill_priority: PRIO=254 2020-03-11T22:51:52.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:254: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:51:52.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:261: TEST_backfill_priority: ceph pg cancel-force-backfill 3.0 2020-03-11T22:51:52.665 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg(s) [3.0] on osd.1 to cancel-force-backfill; 2020-03-11T22:51:52.665 INFO:tasks.workunit.client.0.smithi155.stderr: 2020-03-11T22:51:52.672 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:262: TEST_backfill_priority: sleep 5 2020-03-11T22:51:57.675 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:51:57.676 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:51:57.676 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:51:57.676 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:51:57.676 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:51:57.676 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:51:57.676 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-osd.1.asok 2020-03-11T22:51:57.676 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:51:57.677 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:263: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:51:57.773 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:51:57.773 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:51:57.773 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:57.773 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:57.773 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:51:57.773 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:57.773 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:51:57.774 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:51:57.774 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:57.774 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:51:57.774 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:51:57.774 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:57.774 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:57.774 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:57.775 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:51:57.775 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:57.775 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:51:57.775 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:51:57.775 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:57.775 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:51:57.775 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:51:57.775 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:57.776 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:57.776 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:57.776 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:57.776 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:51:57.776 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:51:57.776 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:51:57.776 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:51:57.776 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:51:57.777 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:51:57.777 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:57.777 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:51:57.777 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:51:57.777 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:51:57.777 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:51:57.777 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:51:57.778 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:51:57.778 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:51:57.778 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:51:57.778 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:51:57.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:266: TEST_backfill_priority: flush_pg_stats 2020-03-11T22:51:57.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:51:57.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:51:58.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:51:58.173 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:51:58.173 INFO:tasks.workunit.client.0.smithi155.stderr:2 2020-03-11T22:51:58.174 INFO:tasks.workunit.client.0.smithi155.stderr:3 2020-03-11T22:51:58.174 INFO:tasks.workunit.client.0.smithi155.stderr:4' 2020-03-11T22:51:58.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:51:58.174 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:58.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:51:58.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836526 2020-03-11T22:51:58.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836526' 2020-03-11T22:51:58.301 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:58.301 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:51:58.434 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705709 2020-03-11T22:51:58.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836526 1-38654705709' 2020-03-11T22:51:58.435 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:58.435 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:51:58.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542188 2020-03-11T22:51:58.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836526 1-38654705709 2-60129542188' 2020-03-11T22:51:58.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:58.562 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:51:58.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378667 2020-03-11T22:51:58.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836526 1-38654705709 2-60129542188 3-81604378667' 2020-03-11T22:51:58.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:51:58.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:51:58.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215145 2020-03-11T22:51:58.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836526 1-38654705709 2-60129542188 3-81604378667 4-103079215145' 2020-03-11T22:51:58.813 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:51:58.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836526 2020-03-11T22:51:58.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:51:58.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:51:58.814 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836526 2020-03-11T22:51:58.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:51:58.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836526 2020-03-11T22:51:58.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836526' 2020-03-11T22:51:58.816 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:51:58.816 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836526 2020-03-11T22:51:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836524 -lt 21474836526 2020-03-11T22:51:59.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:52:00.160 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:52:00.161 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:52:00.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836527 -lt 21474836526 2020-03-11T22:52:00.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:00.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705709 2020-03-11T22:52:00.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:00.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:52:00.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705709 2020-03-11T22:52:00.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:00.531 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705709 2020-03-11T22:52:00.532 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705709' 2020-03-11T22:52:00.532 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:52:00.532 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705709 2020-03-11T22:52:00.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705709 -lt 38654705709 2020-03-11T22:52:00.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:00.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542188 2020-03-11T22:52:00.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:00.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:52:00.883 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542188 2020-03-11T22:52:00.884 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:00.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542188 2020-03-11T22:52:00.885 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542188' 2020-03-11T22:52:00.885 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:52:00.885 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542188 2020-03-11T22:52:01.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542188 -lt 60129542188 2020-03-11T22:52:01.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:01.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378667 2020-03-11T22:52:01.239 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:01.239 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:52:01.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378667 2020-03-11T22:52:01.240 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:01.240 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378667 2020-03-11T22:52:01.241 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378667' 2020-03-11T22:52:01.241 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:52:01.242 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 81604378667 2020-03-11T22:52:01.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378667 -lt 81604378667 2020-03-11T22:52:01.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:01.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215145 2020-03-11T22:52:01.589 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:01.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:52:01.590 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215145 2020-03-11T22:52:01.590 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:01.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215145 2020-03-11T22:52:01.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215145' 2020-03-11T22:52:01.591 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:52:01.592 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.4 seq 103079215145 2020-03-11T22:52:01.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215146 -lt 103079215145 2020-03-11T22:52:01.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:267: TEST_backfill_priority: ceph pg force-backfill 7.0 2020-03-11T22:52:02.261 INFO:tasks.workunit.client.0.smithi155.stderr:instructing pg(s) [7.0] on osd.1 to force-backfill; 2020-03-11T22:52:02.261 INFO:tasks.workunit.client.0.smithi155.stderr: 2020-03-11T22:52:02.269 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:268: TEST_backfill_priority: sleep 2 2020-03-11T22:52:04.271 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:52:04.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:52:04.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:52:04.272 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:04.272 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:04.272 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:52:04.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-osd.1.asok 2020-03-11T22:52:04.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:52:04.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:270: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:04.377 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:271: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:52:04.378 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:04.378 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:04.378 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:04.378 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:04.379 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:04.379 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:04.379 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:04.379 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:04.379 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:04.379 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:04.379 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:04.380 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:04.380 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:04.380 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:04.380 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:04.380 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:04.380 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:52:04.380 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:04.381 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:04.381 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:04.381 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:04.381 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:04.381 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:04.381 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:04.381 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:04.382 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:04.382 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:04.382 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:04.382 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:52:04.382 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:52:04.382 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:04.382 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:04.382 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:04.383 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:04.383 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:04.383 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:04.383 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:04.383 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:04.383 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:04.383 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:04.384 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:04.385 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:52:04.385 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: jq '(.local_reservations.queues[].items[] | select(.item == "3.0")).prio' 2020-03-11T22:52:04.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:272: TEST_backfill_priority: PRIO=151 2020-03-11T22:52:04.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:273: TEST_backfill_priority: '[' 151 '!=' 151 ']' 2020-03-11T22:52:04.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:52:04.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: jq '.local_reservations.in_progress[0].item' 2020-03-11T22:52:04.386 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: eval 'ITEM="7.0"' 2020-03-11T22:52:04.386 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:279: TEST_backfill_priority: ITEM=7.0 2020-03-11T22:52:04.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:280: TEST_backfill_priority: '[' 7.0 '!=' 7.0 ']' 2020-03-11T22:52:04.387 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: cat td/osd-backfill-prio/out 2020-03-11T22:52:04.387 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: jq '.local_reservations.in_progress[0].prio' 2020-03-11T22:52:04.387 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:285: TEST_backfill_priority: PRIO=254 2020-03-11T22:52:04.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:286: TEST_backfill_priority: '[' 254 '!=' 254 ']' 2020-03-11T22:52:04.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:293: TEST_backfill_priority: ceph osd unset noout 2020-03-11T22:52:05.000 INFO:tasks.workunit.client.0.smithi155.stderr:noout is unset 2020-03-11T22:52:05.008 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:294: TEST_backfill_priority: ceph osd unset nobackfill 2020-03-11T22:52:05.678 INFO:tasks.workunit.client.0.smithi155.stderr:nobackfill is unset 2020-03-11T22:52:05.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:52:05.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:52:05.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:52:05.687 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:52:05.687 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:52:05.687 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:52:05.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-osd.1.asok 2020-03-11T22:52:05.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:296: TEST_backfill_priority: wait_for_clean 'CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:52:05.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local 'cmd=CEPH_ARGS='\'''\'' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations' 2020-03-11T22:52:05.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:52:05.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:52:05.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:52:05.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:52:05.689 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:52:05.690 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:52:05.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:52:05.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:52:05.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:52:05.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:52:05.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:52:05.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:52:05.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:52:05.780 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:52:06.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:52:06.207 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:52:06.208 INFO:tasks.workunit.client.0.smithi155.stderr:2 2020-03-11T22:52:06.208 INFO:tasks.workunit.client.0.smithi155.stderr:3 2020-03-11T22:52:06.208 INFO:tasks.workunit.client.0.smithi155.stderr:4' 2020-03-11T22:52:06.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:52:06.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:06.208 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:52:06.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836530 2020-03-11T22:52:06.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530' 2020-03-11T22:52:06.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:06.438 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:52:06.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705713 2020-03-11T22:52:06.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530 1-38654705713' 2020-03-11T22:52:06.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:06.598 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:52:06.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=60129542191 2020-03-11T22:52:06.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530 1-38654705713 2-60129542191' 2020-03-11T22:52:06.724 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:06.725 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:52:06.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=81604378670 2020-03-11T22:52:06.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530 1-38654705713 2-60129542191 3-81604378670' 2020-03-11T22:52:06.853 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:52:06.854 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:52:06.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=103079215149 2020-03-11T22:52:06.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836530 1-38654705713 2-60129542191 3-81604378670 4-103079215149' 2020-03-11T22:52:06.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:06.982 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836530 2020-03-11T22:52:06.982 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:06.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:52:06.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836530 2020-03-11T22:52:06.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:06.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836530 2020-03-11T22:52:06.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836530' 2020-03-11T22:52:06.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:52:06.984 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836530 2020-03-11T22:52:07.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 0 -lt 21474836530 2020-03-11T22:52:07.382 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:52:08.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:52:08.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:52:08.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836530 -lt 21474836530 2020-03-11T22:52:08.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:08.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705713 2020-03-11T22:52:08.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:08.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:52:08.810 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705713 2020-03-11T22:52:08.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:08.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705713 2020-03-11T22:52:08.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705713' 2020-03-11T22:52:08.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:52:08.812 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705713 2020-03-11T22:52:09.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705713 -lt 38654705713 2020-03-11T22:52:09.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:09.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-60129542191 2020-03-11T22:52:09.165 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:09.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:52:09.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-60129542191 2020-03-11T22:52:09.166 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:09.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=60129542191 2020-03-11T22:52:09.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 60129542191' 2020-03-11T22:52:09.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:52:09.168 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 60129542191 2020-03-11T22:52:09.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 60129542192 -lt 60129542191 2020-03-11T22:52:09.587 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:09.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-81604378670 2020-03-11T22:52:09.587 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:09.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:52:09.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-81604378670 2020-03-11T22:52:09.588 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:09.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=81604378670 2020-03-11T22:52:09.589 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 81604378670' 2020-03-11T22:52:09.589 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 81604378670 2020-03-11T22:52:09.590 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:52:09.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 81604378670 -lt 81604378670 2020-03-11T22:52:09.990 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:52:09.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-103079215149 2020-03-11T22:52:09.991 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:52:09.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:52:09.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-103079215149 2020-03-11T22:52:09.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:52:09.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=103079215149 2020-03-11T22:52:09.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 103079215149' 2020-03-11T22:52:09.993 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:52:09.993 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.4 seq 103079215149 2020-03-11T22:52:10.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 103079215149 -lt 103079215149 2020-03-11T22:52:10.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:52:10.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:10.430 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:10.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 3 == 0 2020-03-11T22:52:10.880 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:10.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:10.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:10.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:10.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:10.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:10.881 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:11.222 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:11.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:11.222 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:11.223 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:11.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:11.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:52:11.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:52:11.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:52:11.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:11.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:11.674 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:11.761 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:11.761 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:11.761 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:11.761 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:11.761 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:11.761 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:11.762 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:11.762 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:11.762 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:11.762 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:11.762 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:11.762 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:11.762 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:11.763 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:11.763 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:11.763 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:11.763 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:52:11.763 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:11.763 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:11.763 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:11.764 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:11.764 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:11.764 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:11.764 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:11.764 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:11.764 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:11.764 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:11.764 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:11.765 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:52:11.765 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:52:11.765 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:11.765 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:11.765 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:11.765 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:11.765 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:11.766 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:11.766 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:11.766 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:11.766 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:11.766 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:11.766 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:11.767 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:11.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:11.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:11.870 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:11.870 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:11.871 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:11.871 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:11.871 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:11.871 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:12.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:12.210 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:12.210 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:12.210 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:12.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:12.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:12.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:12.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:12.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:12.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:12.661 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:12.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:12.662 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:13.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17298803 2020-03-11T22:52:13.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17298803 '!=' null 2020-03-11T22:52:13.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:13.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:13.114 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:13.115 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:13.202 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:13.203 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:13.203 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:13.203 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:13.203 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:13.203 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:13.203 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:13.203 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:13.203 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:13.204 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:13.204 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:13.204 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:13.204 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:13.204 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:13.204 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:13.204 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:13.205 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:52:13.205 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:13.205 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:13.205 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:13.205 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:13.205 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:13.205 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:13.205 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:13.206 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:13.206 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:13.206 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:13.206 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:13.206 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:52:13.206 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:52:13.206 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:13.207 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:13.207 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:13.207 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:13.207 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:13.207 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:13.207 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:13.207 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:13.207 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:13.208 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:13.208 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:13.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:13.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:13.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:13.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:13.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:13.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:13.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:13.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:13.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:13.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:13.650 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:13.651 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:13.651 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:14.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:14.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:14.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:14.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:14.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:14.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:14.101 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:14.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:14.101 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:14.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14207273 2020-03-11T22:52:14.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14207273 '!=' null 2020-03-11T22:52:14.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:14.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:14.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:14.596 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:14.684 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:14.684 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:14.684 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:14.684 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:14.684 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:14.684 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:14.685 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:14.685 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:14.685 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:14.685 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:14.685 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:14.685 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:14.685 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:14.685 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:14.686 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:14.686 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:14.686 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:52:14.686 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:14.686 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:14.686 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:14.686 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:14.687 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:14.687 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:14.687 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:14.687 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:14.687 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:14.687 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:14.687 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:14.688 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:52:14.688 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:52:14.688 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:14.688 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:14.688 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:14.688 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:14.688 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:14.688 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:14.689 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:14.689 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:14.689 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:14.689 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:14.689 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:14.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:14.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:14.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:14.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:14.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:14.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:14.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:14.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:14.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:15.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:15.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:15.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:15.132 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:15.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:15.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:15.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:15.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:15.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:15.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:15.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:15.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:15.584 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:16.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14207273 2020-03-11T22:52:16.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14207273 '!=' null 2020-03-11T22:52:16.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:16.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:16.037 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:16.038 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:16.126 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:16.126 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:16.127 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:16.127 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:16.127 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:16.127 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:16.127 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:16.127 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:16.127 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:16.128 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:16.128 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:16.128 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:16.128 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:16.128 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:16.128 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:16.128 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:16.129 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:52:16.129 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:16.129 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:16.129 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:16.129 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:16.129 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:16.129 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:16.130 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:16.130 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:16.130 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:16.130 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:16.130 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:16.130 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:52:16.130 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:52:16.130 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:16.131 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:16.131 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:16.131 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:16.131 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:16.131 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:16.131 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:16.131 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:16.131 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:16.132 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:16.132 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:16.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:16.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:16.235 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:16.235 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:16.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:16.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:16.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:16.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:16.236 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:16.579 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:16.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:16.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:16.580 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:17.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:17.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:17.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:17.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:17.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:17.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:17.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:17.028 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:17.028 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:17.557 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23746862 2020-03-11T22:52:17.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23746862 '!=' null 2020-03-11T22:52:17.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:17.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:17.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:17.558 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:17.646 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:17.646 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:17.646 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:17.646 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:17.646 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:17.646 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:17.646 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:17.647 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:52:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:17.648 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:52:17.649 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:52:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:17.650 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:17.651 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:17.651 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:17.651 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:17.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:17.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:17.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:17.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:17.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:17.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:17.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:17.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:17.755 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:18.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:18.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:18.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:18.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:18.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:18.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:18.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:18.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:18.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:18.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:18.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:18.610 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:18.610 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:19.282 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967239 2020-03-11T22:52:19.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967239 '!=' null 2020-03-11T22:52:19.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:19.283 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:19.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:19.283 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:19.373 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:19.373 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:19.373 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:19.373 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:19.373 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:19.374 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:19.374 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:19.374 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:19.374 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:19.374 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:19.374 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:19.375 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:19.375 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:19.375 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:19.375 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:19.375 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:19.376 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:52:19.376 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:19.376 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:19.376 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:19.376 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:19.376 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:19.377 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:19.377 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:19.377 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:19.377 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:19.377 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:19.377 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:19.378 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:52:19.378 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:52:19.378 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:19.378 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:19.378 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:19.379 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:19.379 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:19.379 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:19.379 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:19.380 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:19.380 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:19.380 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:19.381 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:19.381 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:19.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:19.481 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:19.482 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:19.482 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:19.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:19.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:19.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:19.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:19.824 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:19.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:19.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:19.824 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:20.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:20.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:20.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:20.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:20.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:20.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:20.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:20.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:20.281 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:20.815 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20093676 2020-03-11T22:52:20.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20093676 '!=' null 2020-03-11T22:52:20.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:20.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:20.816 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:20.816 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:20.904 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:20.904 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:20.904 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:20.904 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:20.905 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:20.905 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:20.905 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:20.905 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:20.905 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:20.905 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:20.905 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:20.905 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:20.906 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:20.906 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:20.906 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:20.906 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:20.906 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:52:20.906 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:20.906 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:20.907 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:20.907 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:20.907 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:20.907 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:20.907 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:20.907 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:20.907 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:20.907 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:20.908 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:20.908 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:52:20.908 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:52:20.908 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:20.908 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:20.908 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:20.908 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:20.909 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:20.909 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:20.909 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:20.909 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:20.909 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:20.909 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:20.909 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:20.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:21.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:21.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:21.013 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:21.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:21.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:21.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:21.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:21.014 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:21.351 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:21.352 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:21.352 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:21.352 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:21.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:21.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:21.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:21.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:21.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:21.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:21.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:21.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:21.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:22.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20093676 2020-03-11T22:52:22.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20093676 '!=' null 2020-03-11T22:52:22.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:22.367 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:22.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:22.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:22.454 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:22.454 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:22.454 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:22.454 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:22.455 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:22.455 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:22.455 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:22.455 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:22.455 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:22.456 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:22.456 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:22.456 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:22.456 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:22.456 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:22.457 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:22.457 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:22.457 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:52:22.457 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:22.458 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:22.458 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:22.458 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:22.458 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:22.458 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:22.459 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:22.459 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:22.459 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:22.459 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:22.459 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:22.460 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:52:22.460 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:52:22.460 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:22.460 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:22.460 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:22.461 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:22.461 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:22.461 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:22.461 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:22.461 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:22.461 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:22.462 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:22.462 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:22.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:22.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:22.563 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:22.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:22.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:22.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:22.563 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:22.564 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:22.564 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:22.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:22.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:22.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:22.904 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:23.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:23.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:23.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:23.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:23.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:23.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:23.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:23.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:23.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:23.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451702 2020-03-11T22:52:23.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451702 '!=' null 2020-03-11T22:52:23.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:23.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:23.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:23.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:23.939 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:23.939 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:23.939 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:23.939 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:23.939 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:23.940 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:23.940 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:23.940 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:23.940 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:23.940 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:23.940 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:23.940 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:23.941 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:23.941 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:23.941 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:23.941 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:23.941 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:52:23.941 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:23.941 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:23.941 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:23.942 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:23.942 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:23.942 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:23.942 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:23.942 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:23.942 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:23.942 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:23.943 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:23.943 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:52:23.943 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:52:23.943 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:23.943 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:23.943 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:23.943 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:23.943 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:23.944 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:23.944 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:23.944 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:23.944 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:23.944 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:23.944 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:23.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:24.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:24.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:24.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:24.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:24.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:24.048 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:24.049 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:24.049 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:24.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:24.388 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:24.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:24.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:24.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:24.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:24.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:24.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:24.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:24.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:24.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:24.839 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:24.839 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:25.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841222 2020-03-11T22:52:25.296 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841222 '!=' null 2020-03-11T22:52:25.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:25.297 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:25.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:25.297 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:25.385 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:25.385 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:25.385 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:25.385 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:25.385 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:25.385 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:25.386 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:25.386 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:25.386 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:25.386 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:25.386 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:25.386 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:25.386 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:25.387 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:25.387 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:25.387 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:25.387 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 151, 2020-03-11T22:52:25.387 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:25.387 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:25.387 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:25.387 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:25.388 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:25.388 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:25.388 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:25.388 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:25.388 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:25.388 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:25.388 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:25.389 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "7.0", 2020-03-11T22:52:25.389 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 254, 2020-03-11T22:52:25.389 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:25.389 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:25.389 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:25.389 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:25.389 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:25.390 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:25.390 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:25.390 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:25.390 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:25.390 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:25.390 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:25.392 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:25.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:25.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:25.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:25.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:25.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:25.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:25.495 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:25.495 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:25.829 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:25.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:25.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:25.830 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:26.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:26.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:26.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:26.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:26.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:26.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:26.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:26.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:26.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:26.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451267 2020-03-11T22:52:26.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451267 '!=' null 2020-03-11T22:52:26.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:26.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:26.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:26.787 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:26.874 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:26.874 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:26.874 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:26.874 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:26.874 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:26.875 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:26.876 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:26.876 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:26.876 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:26.876 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:26.876 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:26.876 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:26.876 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:26.877 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:26.877 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:26.877 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:26.877 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:26.877 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:26.877 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:26.877 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:26.877 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:26.878 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:26.878 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:26.878 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:26.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:26.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:26.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:26.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:26.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:26.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:26.983 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:26.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:26.984 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:27.383 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:27.383 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:27.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:27.384 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:27.995 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:27.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:27.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:27.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:27.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:27.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:27.996 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:27.996 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:27.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:28.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451267 2020-03-11T22:52:28.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451267 '!=' null 2020-03-11T22:52:28.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:28.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:28.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:28.492 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:28.580 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:28.580 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:28.580 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:28.580 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:28.580 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:28.581 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:28.581 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:28.581 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:28.581 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:28.581 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:28.581 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:28.581 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:28.582 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:28.582 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:28.582 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:28.582 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:28.582 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:28.582 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:28.582 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:28.583 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:28.583 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:28.583 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:28.583 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:28.583 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:28.583 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:28.584 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:28.584 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:28.584 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:28.584 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:28.584 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:28.584 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:28.588 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:28.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:28.689 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:28.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:28.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:28.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:28.690 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:28.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:28.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:29.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:29.038 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:29.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:29.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:29.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:29.493 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:29.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:29.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:29.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:29.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:29.494 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:29.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:29.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:29.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31450981 2020-03-11T22:52:29.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31450981 '!=' null 2020-03-11T22:52:29.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:29.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:29.947 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:29.947 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:30.034 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:30.034 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:30.035 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:30.035 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:30.035 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:30.035 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:30.035 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:30.035 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:30.035 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:30.036 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:30.036 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:30.036 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:30.036 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:30.036 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:30.036 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:30.036 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:30.037 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:30.037 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:30.037 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:30.037 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:30.037 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:30.037 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:30.037 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:30.037 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:30.038 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:30.038 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:30.038 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:30.038 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:30.038 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:30.038 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:30.038 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:30.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:30.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:30.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:30.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:30.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:30.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:30.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:30.142 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:30.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:30.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:30.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:30.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:30.483 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:30.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:30.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:30.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:30.960 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:30.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:30.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:30.961 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:30.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:30.961 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:31.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14414857 2020-03-11T22:52:31.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14414857 '!=' null 2020-03-11T22:52:31.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:31.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:31.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:31.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:31.508 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:31.508 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:31.508 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:31.508 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:31.508 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:31.508 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:31.508 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:31.509 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:31.509 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:31.509 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:31.509 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:31.509 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:31.509 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:31.509 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:31.510 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:31.510 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:31.510 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:31.510 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:31.510 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:31.510 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:31.510 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:31.510 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:31.511 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:31.511 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:31.511 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:31.511 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:31.511 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:31.511 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:31.511 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:31.511 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:31.511 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:31.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:31.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:31.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:31.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:31.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:31.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:31.617 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:31.618 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:31.618 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:31.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:52:31.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:31.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:31.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:32.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 3 2020-03-11T22:52:32.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:52:32.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:32.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:32.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:32.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:32.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:32.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:32.407 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:32.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=28830127 2020-03-11T22:52:32.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 28830127 '!=' null 2020-03-11T22:52:32.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:32.919 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:32.920 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:32.920 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:33.007 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:33.007 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:33.007 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:33.008 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:33.008 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:33.008 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:33.008 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:33.008 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:33.008 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:33.008 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:33.009 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:33.009 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:33.009 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:33.009 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:33.009 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:33.009 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:33.009 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:33.009 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:33.010 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:33.010 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:33.010 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:33.010 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:33.010 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:33.010 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:33.010 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:33.010 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:33.011 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:33.011 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:33.011 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:33.011 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:33.011 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:33.014 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:33.115 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:33.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:33.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:33.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:33.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:33.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:33.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:33.117 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:33.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:33.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:33.457 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:33.458 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:33.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:33.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 0 2020-03-11T22:52:33.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:52:33.909 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=1 2020-03-11T22:52:33.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:33.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:33.910 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:33.997 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:33.997 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:33.998 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:33.999 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:34.000 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:34.001 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:34.001 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:34.004 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:34.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:34.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:34.106 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:34.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:34.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:34.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:34.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:34.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:34.444 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:34.444 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:34.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:34.445 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:34.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:34.953 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:34.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:34.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:34.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:34.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:34.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:34.954 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:34.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:35.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415173 2020-03-11T22:52:35.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415173 '!=' null 2020-03-11T22:52:35.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:35.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:35.401 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:35.401 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:35.490 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:35.490 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:35.490 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:35.490 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:35.491 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:35.491 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:35.491 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:35.491 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:35.491 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:35.491 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:35.491 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:35.492 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:35.492 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:35.492 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:35.492 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:35.492 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:35.492 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:35.492 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:35.492 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:35.493 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:35.493 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:35.493 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:35.493 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:35.493 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:35.493 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:35.493 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:35.494 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:35.494 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:35.494 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:35.494 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:35.494 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:35.497 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:35.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:35.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:35.599 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:35.599 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:35.599 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:35.599 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:35.599 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:35.599 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:35.942 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:35.943 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:35.943 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:35.943 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:36.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:36.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:36.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:36.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:36.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:36.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:36.401 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:36.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:36.402 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:36.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27069512 2020-03-11T22:52:36.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27069512 '!=' null 2020-03-11T22:52:36.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:36.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:36.951 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:36.952 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:37.041 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:37.041 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:37.041 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:37.041 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:37.041 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:37.042 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:37.042 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:37.042 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:37.042 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:37.042 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:37.042 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:37.042 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:37.042 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:37.043 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:37.043 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:37.043 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:37.043 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:37.043 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:37.043 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:37.043 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:37.043 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:37.044 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:37.044 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:37.044 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:37.044 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:37.044 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:37.044 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:37.044 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:37.044 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:37.045 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:37.045 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:37.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:37.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:37.150 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:37.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:37.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:37.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:37.150 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:37.151 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:37.151 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:37.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:37.489 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:37.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:37.490 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:37.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:37.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:37.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:37.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:37.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:37.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:37.941 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:37.942 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:37.942 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:38.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27069512 2020-03-11T22:52:38.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27069512 '!=' null 2020-03-11T22:52:38.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:38.520 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:38.520 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:38.521 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:38.610 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:38.610 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:38.610 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:38.611 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:38.611 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:38.611 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:38.611 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:38.611 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:38.611 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:38.611 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:38.611 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:38.612 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:38.612 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:38.612 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:38.612 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:38.612 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:38.612 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:38.612 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:38.613 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:38.613 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:38.613 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:38.613 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:38.613 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:38.613 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:38.613 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:38.613 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:38.614 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:38.614 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:38.614 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:38.614 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:38.614 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:38.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:38.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:38.718 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:38.718 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:38.718 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:38.719 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:38.719 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:38.719 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:38.719 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:39.094 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:39.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:39.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:39.095 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:39.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:39.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:39.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:39.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:39.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:39.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:39.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:39.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:39.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:40.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23079016 2020-03-11T22:52:40.134 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23079016 '!=' null 2020-03-11T22:52:40.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:40.135 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:40.135 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:40.135 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:40.224 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:40.224 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:40.224 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:40.225 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:40.225 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:40.225 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:40.225 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:40.225 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:40.225 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:40.225 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:40.226 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:40.226 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:40.226 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:40.226 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:40.226 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:40.226 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:40.226 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:40.227 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:40.227 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:40.227 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:40.227 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:40.227 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:40.227 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:40.227 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:40.228 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:40.228 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:40.228 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:40.228 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:40.228 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:40.228 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:40.228 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:40.231 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:40.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:40.332 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:40.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:40.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:40.333 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:40.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:40.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:40.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:40.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:40.671 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:40.672 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:40.672 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:41.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:41.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:41.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:41.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:41.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:41.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:41.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:41.126 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:41.126 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:41.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20967804 2020-03-11T22:52:41.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20967804 '!=' null 2020-03-11T22:52:41.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:41.601 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:41.601 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:41.602 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:41.691 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:41.691 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:41.692 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:41.692 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:41.692 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:41.692 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:41.692 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:41.692 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:41.692 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:41.693 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:41.693 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:41.693 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:41.693 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:41.693 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:41.693 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:41.693 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:41.693 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:41.694 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:41.694 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:41.694 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:41.694 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:41.694 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:41.694 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:41.694 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:41.695 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:41.695 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:41.695 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:41.695 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:41.695 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:41.695 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:41.695 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:41.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:41.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:41.800 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:41.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:41.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:41.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:41.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:41.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:41.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:42.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:42.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:42.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:42.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:42.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:42.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:42.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:42.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:42.596 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:42.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:42.597 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:42.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:42.597 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:43.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31451872 2020-03-11T22:52:43.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31451872 '!=' null 2020-03-11T22:52:43.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:43.058 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:43.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:43.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:43.147 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:43.147 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:43.147 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:43.147 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:43.147 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:43.148 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:43.148 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:43.148 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:43.148 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:43.148 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:43.148 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:43.148 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:43.149 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:43.149 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:43.149 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:43.149 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:43.149 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:43.149 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:43.149 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:43.149 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:43.150 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:43.150 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:43.150 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:43.150 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:43.150 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:43.150 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:43.150 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:43.151 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:43.151 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:43.151 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:43.151 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:43.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:43.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:43.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:43.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:43.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:43.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:43.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:43.256 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:43.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:43.598 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:43.599 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:43.599 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:43.599 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:44.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:44.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:44.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:44.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:44.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:44.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:44.127 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:44.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:44.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:44.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841538 2020-03-11T22:52:44.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841538 '!=' null 2020-03-11T22:52:44.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:44.653 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:44.653 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:44.653 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:44.741 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:44.741 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:44.741 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:44.742 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:44.742 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:44.742 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:44.742 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:44.742 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:44.742 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:44.742 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:44.743 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:44.743 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:44.743 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:44.743 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:44.743 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:44.743 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:44.743 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:44.743 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:44.744 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:44.744 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:44.744 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:44.744 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:44.744 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:44.744 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:44.744 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:44.745 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:44.745 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:44.745 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:44.745 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:44.745 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:44.745 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:44.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:44.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:44.851 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:44.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:44.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:44.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:44.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:44.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:44.851 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:45.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:45.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:45.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:45.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:45.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:45.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:45.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:45.640 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:45.641 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:46.145 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=21841538 2020-03-11T22:52:46.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 21841538 '!=' null 2020-03-11T22:52:46.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:46.146 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:46.146 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:46.147 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:46.233 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:46.234 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:46.234 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:46.234 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:46.234 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [ 2020-03-11T22:52:46.235 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:46.235 INFO:tasks.workunit.client.0.smithi155.stdout: "priority": 110, 2020-03-11T22:52:46.235 INFO:tasks.workunit.client.0.smithi155.stdout: "items": [ 2020-03-11T22:52:46.235 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:46.236 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:46.236 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:46.236 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:46.236 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:46.236 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:46.237 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:46.237 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:52:46.237 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:46.237 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:46.237 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "3.0", 2020-03-11T22:52:46.238 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 151, 2020-03-11T22:52:46.238 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:46.238 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:46.238 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:46.239 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:46.239 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:46.239 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:46.240 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:46.240 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:46.240 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:46.240 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:46.241 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:46.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:46.342 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:46.343 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:46.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:46.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:46.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:46.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:46.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:46.343 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:46.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:46.680 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:46.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:46.681 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:47.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:47.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:47.142 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:47.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:47.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:47.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:47.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:47.143 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:47.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:47.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33198694 2020-03-11T22:52:47.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33198694 '!=' null 2020-03-11T22:52:47.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:47.619 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:47.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:47.619 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:47.705 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:47.706 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:47.706 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:47.706 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:47.706 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:47.706 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:47.706 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:47.707 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:47.707 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:47.707 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:47.707 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:47.707 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:47.707 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:47.707 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:47.707 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:47.708 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:47.708 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:47.708 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:47.708 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:47.708 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:47.713 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:47.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:47.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:47.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:47.815 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:47.816 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:47.816 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:47.816 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:48.237 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:48.237 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:48.237 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:48.237 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:48.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:48.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:48.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:48.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:48.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:48.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:48.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:48.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:48.693 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:49.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=32761501 2020-03-11T22:52:49.155 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 32761501 '!=' null 2020-03-11T22:52:49.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:49.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:49.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:49.156 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:49.244 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:49.244 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:49.244 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:49.244 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:49.244 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:49.244 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:49.245 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:49.245 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:49.245 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:49.245 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:49.245 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:49.245 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:49.245 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:49.245 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:49.246 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:49.246 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:49.246 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:49.246 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:49.246 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:49.246 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:49.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:49.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:49.352 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:49.353 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:49.353 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:49.353 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:49.353 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:49.353 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:49.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:49.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:49.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:49.846 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:49.846 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:50.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:50.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:50.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:50.298 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:50.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:50.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:50.299 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:50.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:50.299 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:50.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17036013 2020-03-11T22:52:50.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17036013 '!=' null 2020-03-11T22:52:50.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:50.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:50.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:50.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:50.906 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:50.906 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:50.906 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:50.907 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:50.907 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:50.907 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:50.907 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:50.907 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:50.907 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:50.907 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:50.907 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:50.908 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:50.908 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:50.908 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:50.908 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:50.908 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:50.908 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:50.908 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:50.909 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:50.909 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:50.913 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:51.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:51.015 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:51.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:51.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:51.015 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:51.016 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:51.016 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:51.016 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:51.354 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:52:51.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:51.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:51.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:51.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 3 2020-03-11T22:52:51.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 1 '!=' 1 2020-03-11T22:52:51.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:51.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:51.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:51.816 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:51.817 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:51.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:51.817 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:52.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=17036013 2020-03-11T22:52:52.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 17036013 '!=' null 2020-03-11T22:52:52.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:52.270 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:52.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:52.270 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:52.359 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:52.359 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:52.360 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:52.360 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:52.360 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:52.360 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:52.360 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:52.360 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:52.360 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:52.361 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:52.361 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:52.361 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:52.361 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:52.361 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:52.361 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:52.361 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:52.362 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:52.362 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:52.362 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:52.362 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:52.366 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:52.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:52.467 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:52.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:52.468 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:52.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:52.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:52.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:52.469 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:52.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:52.804 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:52.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:52.805 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:53.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:53.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 1 2020-03-11T22:52:53.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:52:53.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=2 2020-03-11T22:52:53.373 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:53.373 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:53.373 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:53.464 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:53.464 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:53.464 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:53.465 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:53.465 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:53.465 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:53.465 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:53.465 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:53.465 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:53.465 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:53.466 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:53.466 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:53.466 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:53.466 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:53.466 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:53.466 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:53.466 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:53.466 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:53.467 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:53.467 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:53.472 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:53.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:53.573 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:53.574 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:53.574 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:53.574 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:53.574 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:53.574 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:53.574 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:53.915 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:53.916 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:53.916 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:53.916 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:54.414 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:54.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:54.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:54.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:54.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:54.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:54.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:54.415 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:54.416 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:54.886 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415383 2020-03-11T22:52:54.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415383 '!=' null 2020-03-11T22:52:54.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:54.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:54.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:54.887 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:54.974 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:54.974 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:54.975 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:54.975 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:54.975 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:54.975 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:54.975 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:54.975 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:54.975 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:54.976 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:54.976 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:54.976 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:54.976 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:54.976 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:54.976 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:54.976 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:54.977 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:54.977 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:54.977 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:54.977 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:54.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:55.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:55.082 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:55.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:55.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:55.083 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:55.084 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:55.084 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:55.084 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:55.419 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:55.419 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:55.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:55.420 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:55.869 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:55.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:55.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:55.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:55.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:55.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:55.870 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:55.871 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:55.871 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:56.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27464772 2020-03-11T22:52:56.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27464772 '!=' null 2020-03-11T22:52:56.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:56.556 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:56.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:56.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:56.645 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:56.645 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:56.645 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:56.645 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:56.645 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:56.645 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:56.646 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:56.646 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:56.646 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:56.646 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:56.646 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:56.646 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:56.646 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:56.646 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:56.647 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:56.647 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:56.647 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:56.647 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:56.647 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:56.647 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:56.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:56.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:56.753 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:56.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:56.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:56.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:56.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:56.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:56.754 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:57.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:57.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:57.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:57.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:57.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:57.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:57.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:57.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:57.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:57.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:57.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:57.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:57.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:58.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27464772 2020-03-11T22:52:58.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27464772 '!=' null 2020-03-11T22:52:58.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:58.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:58.157 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:58.158 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:58.245 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:58.245 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:58.245 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:58.245 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:58.245 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:58.245 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:58.246 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:58.246 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:58.246 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:58.246 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:58.246 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:58.246 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:58.246 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:58.247 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:58.247 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:58.247 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:58.247 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:58.247 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:58.247 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:58.247 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:58.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:58.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:58.353 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:58.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:58.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:58.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:58.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:58.354 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:58.355 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:52:58.691 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:52:58.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:52:58.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:52:58.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:52:59.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:52:59.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:52:59.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:52:59.140 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:52:59.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:52:59.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:52:59.141 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:52:59.141 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:52:59.141 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:52:59.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=23270849 2020-03-11T22:52:59.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 23270849 '!=' null 2020-03-11T22:52:59.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:52:59.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:59.594 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:52:59.594 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:52:59.681 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:52:59.681 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:52:59.681 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:59.681 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:59.682 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:59.682 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:52:59.682 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:52:59.682 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:52:59.682 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:52:59.682 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:52:59.682 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:59.683 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:52:59.683 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:52:59.683 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:52:59.683 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:52:59.683 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:52:59.683 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:52:59.683 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:52:59.683 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:52:59.684 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:52:59.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:52:59.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:52:59.789 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:52:59.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:52:59.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:52:59.791 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:52:59.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:52:59.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:52:59.792 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:00.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:00.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:00.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:00.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:00.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:00.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:00.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:00.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:00.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:00.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:00.780 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:00.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:00.781 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:01.294 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094177 2020-03-11T22:53:01.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094177 '!=' null 2020-03-11T22:53:01.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:01.295 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:01.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:01.295 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:01.385 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:53:01.385 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:53:01.385 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:01.385 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:01.386 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:01.386 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:53:01.386 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:01.386 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:53:01.386 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:53:01.386 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:53:01.387 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:01.387 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:01.387 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:01.387 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:53:01.387 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:01.388 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:01.388 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:01.388 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:53:01.388 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:01.388 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:53:01.393 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:01.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:01.492 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:01.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:01.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:01.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:01.493 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:01.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:01.494 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:01.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:01.878 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:01.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:01.879 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:02.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:02.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:02.330 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:02.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:02.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:02.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:02.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:02.331 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:02.331 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:02.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704554 2020-03-11T22:53:02.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704554 '!=' null 2020-03-11T22:53:02.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:02.868 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:02.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:02.869 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:02.959 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:53:02.959 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:53:02.959 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:02.959 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:02.959 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:02.960 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:53:02.960 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:02.960 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:53:02.960 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:53:02.960 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:53:02.960 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:02.960 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:02.960 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:02.961 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:53:02.961 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:02.961 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:02.961 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:02.961 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:53:02.961 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:02.962 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:53:02.966 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:03.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:03.067 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:03.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:03.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:03.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:03.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:03.068 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:03.069 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:03.405 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:03.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:03.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:03.406 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:03.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:03.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:03.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:03.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:03.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:03.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:03.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:03.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:03.969 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:04.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704554 2020-03-11T22:53:04.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704554 '!=' null 2020-03-11T22:53:04.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:04.427 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:04.428 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:04.428 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:04.516 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:53:04.516 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:53:04.516 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:04.516 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:04.517 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:04.517 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:53:04.517 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:04.517 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:53:04.517 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:53:04.517 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:53:04.518 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:04.518 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:04.518 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:04.518 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:53:04.518 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:04.518 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:04.518 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:04.518 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:53:04.519 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:04.519 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:53:04.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:04.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:04.624 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:04.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:04.625 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:04.626 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:04.626 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:04.626 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:04.626 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:04.970 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:04.970 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:04.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:04.971 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:05.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:05.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:05.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:05.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:05.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:05.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:05.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:05.516 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:05.516 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:05.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=20094366 2020-03-11T22:53:05.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 20094366 '!=' null 2020-03-11T22:53:05.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:05.965 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:05.965 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:05.965 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:06.054 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:53:06.054 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:53:06.054 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:06.055 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:06.055 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:06.055 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:53:06.055 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:06.055 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:53:06.055 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:53:06.055 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:53:06.055 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:06.056 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:06.056 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:06.056 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:53:06.056 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:06.056 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:06.056 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:06.056 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:53:06.057 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:06.057 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:53:06.061 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:06.162 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:06.163 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:06.163 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:06.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:06.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:06.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:06.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:06.164 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:06.503 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:06.503 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:06.504 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:06.504 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:06.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:06.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:06.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:06.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:06.954 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:06.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:06.955 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:06.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:06.955 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:07.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=29704428 2020-03-11T22:53:07.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 29704428 '!=' null 2020-03-11T22:53:07.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:07.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:07.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:07.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:07.634 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:53:07.634 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:53:07.634 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:07.634 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:07.635 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:07.635 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [ 2020-03-11T22:53:07.635 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:07.635 INFO:tasks.workunit.client.0.smithi155.stdout: "item": "1.0", 2020-03-11T22:53:07.635 INFO:tasks.workunit.client.0.smithi155.stdout: "prio": 110, 2020-03-11T22:53:07.635 INFO:tasks.workunit.client.0.smithi155.stdout: "can_preempt": true 2020-03-11T22:53:07.636 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:07.636 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:07.636 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:07.636 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:53:07.636 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:07.636 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:07.636 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:07.636 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:53:07.637 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:07.637 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:53:07.641 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:07.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:07.745 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:07.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:07.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:07.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:07.746 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:07.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:07.747 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:08.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:08.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:08.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:08.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:08.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:08.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:08.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:08.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:08.537 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:08.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:08.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:08.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:08.538 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:09.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=19220373 2020-03-11T22:53:09.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 19220373 '!=' null 2020-03-11T22:53:09.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:09.090 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:09.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:09.091 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:09.179 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:53:09.179 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:53:09.179 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:09.179 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:09.179 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:09.179 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:53:09.180 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:09.180 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:53:09.180 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:09.180 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:09.180 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:09.180 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:53:09.180 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:09.181 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:53:09.186 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:09.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:09.288 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:09.288 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:09.288 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:09.288 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:09.288 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:09.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:09.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:09.627 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:09.627 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:09.627 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:09.628 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:10.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:10.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:10.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:10.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:10.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:10.078 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:10.079 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:10.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:10.079 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:10.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415305 2020-03-11T22:53:10.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415305 '!=' null 2020-03-11T22:53:10.523 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:10.524 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:10.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:10.524 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:10.610 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:53:10.611 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:53:10.611 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:10.611 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:10.611 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:10.611 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:53:10.611 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:10.611 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:53:10.612 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:10.612 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:10.612 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:10.612 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:53:10.612 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:10.612 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:53:10.618 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:10.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:10.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:10.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:10.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:10.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:10.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:10.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:10.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:11.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=2 2020-03-11T22:53:11.056 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:11.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:11.057 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:11.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 2 = 3 2020-03-11T22:53:11.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 2 '!=' 2 2020-03-11T22:53:11.504 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:53:11.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:53:11.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:53:11.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:53:11.505 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:53:11.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:53:11.505 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:53:11.947 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14415305 2020-03-11T22:53:11.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14415305 '!=' null 2020-03-11T22:53:11.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:53:11.948 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 'CEPH_ARGS='\'''\''' ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:11.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: CEPH_ARGS= 2020-03-11T22:53:11.948 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_recovery_reservations 2020-03-11T22:53:12.035 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:53:12.035 INFO:tasks.workunit.client.0.smithi155.stdout: "local_reservations": { 2020-03-11T22:53:12.035 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:12.036 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:12.036 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:12.036 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:53:12.036 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:12.036 INFO:tasks.workunit.client.0.smithi155.stdout: "remote_reservations": { 2020-03-11T22:53:12.036 INFO:tasks.workunit.client.0.smithi155.stdout: "max_allowed": 1, 2020-03-11T22:53:12.037 INFO:tasks.workunit.client.0.smithi155.stdout: "min_priority": 0, 2020-03-11T22:53:12.037 INFO:tasks.workunit.client.0.smithi155.stdout: "queues": [], 2020-03-11T22:53:12.037 INFO:tasks.workunit.client.0.smithi155.stdout: "in_progress": [] 2020-03-11T22:53:12.037 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:12.037 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:53:12.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:53:12.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:53:12.143 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:53:12.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:53:12.144 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:53:12.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:53:12.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:53:12.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:53:12.145 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:53:12.477 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=3 2020-03-11T22:53:12.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:53:12.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:53:12.478 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:53:12.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 3 = 3 2020-03-11T22:53:12.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:53:12.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:53:12.924 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:298: TEST_backfill_priority: ceph pg dump pgs 2020-03-11T22:53:13.267 INFO:tasks.workunit.client.0.smithi155.stdout:PG_STAT OBJECTS MISSING_ON_PRIMARY DEGRADED MISPLACED UNFOUND BYTES OMAP_BYTES* OMAP_KEYS* LOG DISK_LOG STATE STATE_STAMP VERSION REPORTED UP UP_PRIMARY ACTING ACTING_PRIMARY LAST_SCRUB SCRUB_STAMP LAST_DEEP_SCRUB DEEP_SCRUB_STAMP SNAPTRIMQ_LEN 2020-03-11T22:53:13.267 INFO:tasks.workunit.client.0.smithi155.stdout:7.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:52:27.440543+0000 91'150 116:344 [1,4] 1 [1,4] 1 0'0 2020-03-11T22:49:58.789140+0000 0'0 2020-03-11T22:49:58.789140+0000 0 2020-03-11T22:53:13.267 INFO:tasks.workunit.client.0.smithi155.stdout:1.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:53:09.628673+0000 91'150 117:350 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:49:44.844495+0000 0'0 2020-03-11T22:49:44.844495+0000 0 2020-03-11T22:53:13.267 INFO:tasks.workunit.client.0.smithi155.stdout:3.0 50 0 0 0 0 524288000 0 0 50 50 active+clean 2020-03-11T22:52:48.202643+0000 91'150 116:353 [1,2] 1 [1,2] 1 0'0 2020-03-11T22:49:49.548052+0000 0'0 2020-03-11T22:49:49.548052+0000 0 2020-03-11T22:53:13.267 INFO:tasks.workunit.client.0.smithi155.stdout: 2020-03-11T22:53:13.267 INFO:tasks.workunit.client.0.smithi155.stdout:* NOTE: Omap statistics are gathered during deep scrub and may be inaccurate soon afterwards depending on utilization. See http://docs.ceph.com/docs/master/dev/placement-group/#omap-statistics for further details. 2020-03-11T22:53:13.268 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:53:13.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: get_asok_path osd.1 2020-03-11T22:53:13.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=osd.1 2020-03-11T22:53:13.278 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n osd.1 ']' 2020-03-11T22:53:13.278 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:13.278 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:13.278 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:53:13.279 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.56110/ceph-osd.1.asok 2020-03-11T22:53:13.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: CEPH_ARGS= 2020-03-11T22:53:13.279 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:300: TEST_backfill_priority: ceph --admin-daemon /tmp/ceph-asok.56110/ceph-osd.1.asok dump_pgstate_history 2020-03-11T22:53:13.367 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:53:13.367 INFO:tasks.workunit.client.0.smithi155.stdout: "pgs": [ 2020-03-11T22:53:13.367 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.368 INFO:tasks.workunit.client.0.smithi155.stdout: "pg": "7.0", 2020-03-11T22:53:13.368 INFO:tasks.workunit.client.0.smithi155.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.368 INFO:tasks.workunit.client.0.smithi155.stdout: "history": [ 2020-03-11T22:53:13.368 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.368 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "62", 2020-03-11T22:53:13.368 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.368 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.368 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.369 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:59.231148+0000", 2020-03-11T22:53:13.369 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:59.231160+0000" 2020-03-11T22:53:13.369 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.369 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.369 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.369 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:59.231167+0000", 2020-03-11T22:53:13.370 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:59.269169+0000" 2020-03-11T22:53:13.370 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.370 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.370 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.370 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:59.269169+0000", 2020-03-11T22:53:13.370 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:59.269264+0000" 2020-03-11T22:53:13.370 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.371 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.371 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:13.371 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:59.269265+0000", 2020-03-11T22:53:13.371 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:59.269286+0000" 2020-03-11T22:53:13.371 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.371 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.371 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:13.371 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:59.269287+0000", 2020-03-11T22:53:13.372 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:59.459138+0000" 2020-03-11T22:53:13.372 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.372 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.372 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.372 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:59.231164+0000", 2020-03-11T22:53:13.372 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:59.459149+0000" 2020-03-11T22:53:13.372 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.373 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.373 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:13.373 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:59.459319+0000", 2020-03-11T22:53:13.373 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:59.942117+0000" 2020-03-11T22:53:13.373 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.373 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.373 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:13.374 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:59.942118+0000", 2020-03-11T22:53:13.374 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:59.942179+0000" 2020-03-11T22:53:13.374 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.374 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.374 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.374 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:59.942179+0000", 2020-03-11T22:53:13.374 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:00.927159+0000" 2020-03-11T22:53:13.374 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.375 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.375 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:13.375 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:59.459149+0000", 2020-03-11T22:53:13.375 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:00.927182+0000" 2020-03-11T22:53:13.375 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.376 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.376 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.376 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:59.231160+0000", 2020-03-11T22:53:13.376 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:00.927198+0000" 2020-03-11T22:53:13.376 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.377 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.377 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.377 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:59.231139+0000", 2020-03-11T22:53:13.377 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:00.927203+0000" 2020-03-11T22:53:13.378 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.378 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.378 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.378 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.378 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "63", 2020-03-11T22:53:13.379 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.379 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.379 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.379 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:00.927203+0000", 2020-03-11T22:53:13.379 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:00.927439+0000" 2020-03-11T22:53:13.379 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.380 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.380 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.380 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.380 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "89", 2020-03-11T22:53:13.380 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.381 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.381 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.381 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:00.927443+0000", 2020-03-11T22:53:13.381 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:00.927455+0000" 2020-03-11T22:53:13.381 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.381 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.382 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.382 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:00.927462+0000", 2020-03-11T22:53:13.382 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:01.055730+0000" 2020-03-11T22:53:13.382 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.382 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.383 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.383 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:01.055731+0000", 2020-03-11T22:53:13.383 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:01.055797+0000" 2020-03-11T22:53:13.383 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.383 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.383 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:13.383 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:01.055800+0000", 2020-03-11T22:53:13.384 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:01.055817+0000" 2020-03-11T22:53:13.384 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.384 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.384 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:13.384 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:01.055818+0000", 2020-03-11T22:53:13.384 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:01.229754+0000" 2020-03-11T22:53:13.384 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.385 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.385 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.385 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:00.927458+0000", 2020-03-11T22:53:13.385 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:01.229765+0000" 2020-03-11T22:53:13.385 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.385 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.385 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:13.386 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:01.229891+0000", 2020-03-11T22:53:13.386 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:01.506143+0000" 2020-03-11T22:53:13.386 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.386 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.386 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:13.386 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:01.506143+0000", 2020-03-11T22:53:13.386 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:01.506194+0000" 2020-03-11T22:53:13.387 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.387 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.387 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.387 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:01.506194+0000", 2020-03-11T22:53:13.387 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.251136+0000" 2020-03-11T22:53:13.387 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.387 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.388 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:13.388 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:01.229766+0000", 2020-03-11T22:53:13.388 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.251166+0000" 2020-03-11T22:53:13.388 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.388 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.388 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.388 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:00.927455+0000", 2020-03-11T22:53:13.388 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.251191+0000" 2020-03-11T22:53:13.389 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.389 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.389 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.389 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:00.927440+0000", 2020-03-11T22:53:13.389 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.251200+0000" 2020-03-11T22:53:13.389 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.389 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.390 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.390 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.390 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "90", 2020-03-11T22:53:13.390 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.390 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.390 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.390 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.251200+0000", 2020-03-11T22:53:13.391 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.251601+0000" 2020-03-11T22:53:13.391 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.391 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.391 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.391 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.391 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "96", 2020-03-11T22:53:13.391 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.392 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.392 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.392 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.251608+0000", 2020-03-11T22:53:13.392 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.251631+0000" 2020-03-11T22:53:13.392 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.392 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.392 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.392 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.251643+0000", 2020-03-11T22:53:13.393 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.251824+0000" 2020-03-11T22:53:13.393 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.393 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.393 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.393 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.251824+0000", 2020-03-11T22:53:13.393 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.251906+0000" 2020-03-11T22:53:13.393 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.394 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.394 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:13.394 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.251906+0000", 2020-03-11T22:53:13.394 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.251928+0000" 2020-03-11T22:53:13.394 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.394 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.394 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:13.395 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.251928+0000", 2020-03-11T22:53:13.395 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.561084+0000" 2020-03-11T22:53:13.395 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.395 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.395 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.395 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.251638+0000", 2020-03-11T22:53:13.396 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.561092+0000" 2020-03-11T22:53:13.396 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.396 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.396 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:13.396 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.561163+0000", 2020-03-11T22:53:13.396 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.586450+0000" 2020-03-11T22:53:13.397 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.397 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.397 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:13.397 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.586451+0000", 2020-03-11T22:53:13.397 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:25.586494+0000" 2020-03-11T22:53:13.397 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.397 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.398 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.398 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.586494+0000", 2020-03-11T22:53:13.398 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:24.836720+0000" 2020-03-11T22:53:13.398 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.398 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.398 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:13.399 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.561092+0000", 2020-03-11T22:53:13.399 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:24.836744+0000" 2020-03-11T22:53:13.399 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.399 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.399 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.399 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.251632+0000", 2020-03-11T22:53:13.399 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:24.836785+0000" 2020-03-11T22:53:13.400 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.400 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.400 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.400 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:25.251601+0000", 2020-03-11T22:53:13.400 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:24.836790+0000" 2020-03-11T22:53:13.400 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.400 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.400 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.401 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.401 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "97", 2020-03-11T22:53:13.401 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.401 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.401 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.401 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:24.836791+0000", 2020-03-11T22:53:13.402 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:24.837119+0000" 2020-03-11T22:53:13.402 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.402 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.402 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.402 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.402 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "98", 2020-03-11T22:53:13.402 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.403 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.403 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.403 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:24.837124+0000", 2020-03-11T22:53:13.403 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:24.837137+0000" 2020-03-11T22:53:13.403 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.403 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.403 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.403 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:24.837147+0000", 2020-03-11T22:53:13.404 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:24.857546+0000" 2020-03-11T22:53:13.404 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.404 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.404 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.404 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:24.857546+0000", 2020-03-11T22:53:13.404 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:24.857603+0000" 2020-03-11T22:53:13.404 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.405 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.405 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.405 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:24.837143+0000", 2020-03-11T22:53:13.405 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:24.857613+0000" 2020-03-11T22:53:13.405 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.405 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.406 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:53:13.406 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:24.857614+0000", 2020-03-11T22:53:13.406 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.091572+0000" 2020-03-11T22:53:13.406 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.406 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.406 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.407 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:24.837137+0000", 2020-03-11T22:53:13.407 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.091589+0000" 2020-03-11T22:53:13.407 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.407 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.408 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.408 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:24.837119+0000", 2020-03-11T22:53:13.408 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.091593+0000" 2020-03-11T22:53:13.408 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.408 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.408 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.409 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.409 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "98", 2020-03-11T22:53:13.409 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.409 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.409 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.409 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.091594+0000", 2020-03-11T22:53:13.410 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.091821+0000" 2020-03-11T22:53:13.410 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.410 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.410 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.410 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.410 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "112", 2020-03-11T22:53:13.410 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.410 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.411 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.411 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.091826+0000", 2020-03-11T22:53:13.411 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.091839+0000" 2020-03-11T22:53:13.411 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.411 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.411 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.412 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.091846+0000", 2020-03-11T22:53:13.412 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.130032+0000" 2020-03-11T22:53:13.412 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.412 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.412 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.412 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.130032+0000", 2020-03-11T22:53:13.412 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.130110+0000" 2020-03-11T22:53:13.413 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.413 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.413 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:13.413 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.130110+0000", 2020-03-11T22:53:13.413 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.130130+0000" 2020-03-11T22:53:13.413 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.414 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.414 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:13.414 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.130130+0000", 2020-03-11T22:53:13.414 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.347632+0000" 2020-03-11T22:53:13.414 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.414 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.414 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.415 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.091843+0000", 2020-03-11T22:53:13.415 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.347645+0000" 2020-03-11T22:53:13.415 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.415 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.415 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:13.415 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.347875+0000", 2020-03-11T22:53:13.416 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.391299+0000" 2020-03-11T22:53:13.416 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.416 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.416 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:53:13.416 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.391300+0000", 2020-03-11T22:53:13.416 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.391424+0000" 2020-03-11T22:53:13.416 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.417 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.417 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:53:13.417 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.391424+0000", 2020-03-11T22:53:13.417 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:25.391754+0000" 2020-03-11T22:53:13.417 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.417 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.417 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:53:13.418 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.391755+0000", 2020-03-11T22:53:13.418 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:50.107455+0000" 2020-03-11T22:53:13.418 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.418 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.418 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:53:13.418 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:50.107455+0000", 2020-03-11T22:53:13.418 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:50.107648+0000" 2020-03-11T22:53:13.419 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.419 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.419 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:53:13.419 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:50.107648+0000", 2020-03-11T22:53:13.419 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:02.166088+0000" 2020-03-11T22:53:13.419 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.419 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.420 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:53:13.420 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:02.166088+0000", 2020-03-11T22:53:13.420 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:02.166435+0000" 2020-03-11T22:53:13.420 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.420 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.420 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:53:13.420 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:02.166436+0000", 2020-03-11T22:53:13.421 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:26.777567+0000" 2020-03-11T22:53:13.421 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.421 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.421 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:13.421 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:26.777567+0000", 2020-03-11T22:53:13.421 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:26.777731+0000" 2020-03-11T22:53:13.421 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.422 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.422 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.422 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:26.777732+0000", 2020-03-11T22:53:13.422 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:27.042770+0000" 2020-03-11T22:53:13.422 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.422 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.423 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:13.423 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.347645+0000", 2020-03-11T22:53:13.423 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:27.042838+0000" 2020-03-11T22:53:13.423 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.423 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.423 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.423 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.091839+0000", 2020-03-11T22:53:13.424 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:27.042874+0000" 2020-03-11T22:53:13.424 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.424 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.424 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.424 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:25.091822+0000", 2020-03-11T22:53:13.424 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:27.042886+0000" 2020-03-11T22:53:13.424 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.425 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.425 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.425 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.425 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "112", 2020-03-11T22:53:13.425 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.425 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.426 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.426 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:27.042888+0000", 2020-03-11T22:53:13.426 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:27.043370+0000" 2020-03-11T22:53:13.426 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.426 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.426 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.426 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.427 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.427 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.427 INFO:tasks.workunit.client.0.smithi155.stdout: "pg": "3.0", 2020-03-11T22:53:13.427 INFO:tasks.workunit.client.0.smithi155.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.427 INFO:tasks.workunit.client.0.smithi155.stdout: "history": [ 2020-03-11T22:53:13.427 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.427 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "40", 2020-03-11T22:53:13.427 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.428 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.428 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.428 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:50.452407+0000", 2020-03-11T22:53:13.428 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:50.452416+0000" 2020-03-11T22:53:13.428 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.428 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.428 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.429 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:50.452422+0000", 2020-03-11T22:53:13.429 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:50.525073+0000" 2020-03-11T22:53:13.429 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.429 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.429 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.429 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:50.525073+0000", 2020-03-11T22:53:13.429 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:50.525131+0000" 2020-03-11T22:53:13.429 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.430 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.430 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:13.430 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:50.525131+0000", 2020-03-11T22:53:13.430 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:50.525143+0000" 2020-03-11T22:53:13.430 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.430 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.430 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:13.431 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:50.525143+0000", 2020-03-11T22:53:13.431 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:50.706144+0000" 2020-03-11T22:53:13.431 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.431 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.431 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.431 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:50.452419+0000", 2020-03-11T22:53:13.431 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:50.706155+0000" 2020-03-11T22:53:13.432 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.432 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.432 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:13.432 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:50.706360+0000", 2020-03-11T22:53:13.432 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:50.830166+0000" 2020-03-11T22:53:13.432 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.432 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.432 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:13.433 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:50.830167+0000", 2020-03-11T22:53:13.433 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:50.830248+0000" 2020-03-11T22:53:13.433 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.433 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.433 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.433 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:50.830249+0000", 2020-03-11T22:53:13.433 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.253126+0000" 2020-03-11T22:53:13.434 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.434 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.434 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:13.434 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:50.706155+0000", 2020-03-11T22:53:13.434 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.253160+0000" 2020-03-11T22:53:13.434 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.434 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.434 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.435 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:50.452416+0000", 2020-03-11T22:53:13.435 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.253186+0000" 2020-03-11T22:53:13.435 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.435 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.435 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.435 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:50.452403+0000", 2020-03-11T22:53:13.435 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.253196+0000" 2020-03-11T22:53:13.436 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.436 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.436 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.436 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.436 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "40", 2020-03-11T22:53:13.436 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.436 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.436 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.437 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.253197+0000", 2020-03-11T22:53:13.437 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.253555+0000" 2020-03-11T22:53:13.437 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.437 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.437 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.437 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.437 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "87", 2020-03-11T22:53:13.438 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.438 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.438 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.438 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.253564+0000", 2020-03-11T22:53:13.438 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.253588+0000" 2020-03-11T22:53:13.438 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.438 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.439 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.439 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.253604+0000", 2020-03-11T22:53:13.439 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.353795+0000" 2020-03-11T22:53:13.439 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.439 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.439 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.439 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.353795+0000", 2020-03-11T22:53:13.440 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.353860+0000" 2020-03-11T22:53:13.440 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.440 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.440 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:13.440 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.353861+0000", 2020-03-11T22:53:13.440 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.353875+0000" 2020-03-11T22:53:13.440 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.440 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.441 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:13.441 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.353876+0000", 2020-03-11T22:53:13.441 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.551754+0000" 2020-03-11T22:53:13.441 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.441 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.441 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.441 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.253597+0000", 2020-03-11T22:53:13.442 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.551764+0000" 2020-03-11T22:53:13.442 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.442 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.442 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:13.442 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.551898+0000", 2020-03-11T22:53:13.442 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.700655+0000" 2020-03-11T22:53:13.442 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.442 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.443 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:13.443 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.700656+0000", 2020-03-11T22:53:13.443 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:51.700703+0000" 2020-03-11T22:53:13.443 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.443 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.443 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.443 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.700703+0000", 2020-03-11T22:53:13.444 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.101577+0000" 2020-03-11T22:53:13.444 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.444 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.444 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:13.444 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.551764+0000", 2020-03-11T22:53:13.444 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.101597+0000" 2020-03-11T22:53:13.444 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.445 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.445 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.445 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.253589+0000", 2020-03-11T22:53:13.445 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.101610+0000" 2020-03-11T22:53:13.445 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.445 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.445 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.445 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:51.253555+0000", 2020-03-11T22:53:13.446 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.101615+0000" 2020-03-11T22:53:13.446 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.446 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.446 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.446 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.446 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "87", 2020-03-11T22:53:13.446 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.447 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.447 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.447 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.101615+0000", 2020-03-11T22:53:13.447 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.101796+0000" 2020-03-11T22:53:13.447 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.447 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.447 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.447 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.448 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "103", 2020-03-11T22:53:13.448 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.448 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.448 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.448 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.101801+0000", 2020-03-11T22:53:13.448 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.101813+0000" 2020-03-11T22:53:13.448 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.449 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.449 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.449 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.101821+0000", 2020-03-11T22:53:13.449 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.118197+0000" 2020-03-11T22:53:13.449 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.449 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.450 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.450 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.118197+0000", 2020-03-11T22:53:13.450 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.118292+0000" 2020-03-11T22:53:13.450 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.450 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.450 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:13.450 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.118292+0000", 2020-03-11T22:53:13.451 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.118315+0000" 2020-03-11T22:53:13.451 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.451 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.451 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:13.451 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.118315+0000", 2020-03-11T22:53:13.451 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.434092+0000" 2020-03-11T22:53:13.451 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.452 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.452 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.452 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.101817+0000", 2020-03-11T22:53:13.452 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.434101+0000" 2020-03-11T22:53:13.452 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.452 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.452 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:13.452 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.434223+0000", 2020-03-11T22:53:13.453 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.442849+0000" 2020-03-11T22:53:13.453 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.453 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.453 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:13.453 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.442850+0000", 2020-03-11T22:53:13.453 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:50:24.442894+0000" 2020-03-11T22:53:13.453 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.454 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.454 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.454 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.442895+0000", 2020-03-11T22:53:13.454 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.144957+0000" 2020-03-11T22:53:13.454 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.454 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.454 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:13.455 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.434103+0000", 2020-03-11T22:53:13.455 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.144990+0000" 2020-03-11T22:53:13.455 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.455 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.455 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.455 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.101813+0000", 2020-03-11T22:53:13.456 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.145036+0000" 2020-03-11T22:53:13.456 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.456 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.456 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.456 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:50:24.101797+0000", 2020-03-11T22:53:13.456 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.145042+0000" 2020-03-11T22:53:13.457 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.457 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.457 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.457 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.457 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "104", 2020-03-11T22:53:13.457 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.457 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.457 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.458 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.145042+0000", 2020-03-11T22:53:13.458 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.145398+0000" 2020-03-11T22:53:13.458 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.458 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.458 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.458 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.458 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "106", 2020-03-11T22:53:13.459 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.459 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.459 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.459 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.145403+0000", 2020-03-11T22:53:13.459 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.145417+0000" 2020-03-11T22:53:13.459 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.459 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.459 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.460 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.145427+0000", 2020-03-11T22:53:13.460 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.274758+0000" 2020-03-11T22:53:13.460 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.460 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.460 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.460 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.274758+0000", 2020-03-11T22:53:13.460 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.274815+0000" 2020-03-11T22:53:13.461 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.461 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.461 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.461 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.145423+0000", 2020-03-11T22:53:13.461 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.274825+0000" 2020-03-11T22:53:13.461 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.461 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.462 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:53:13.462 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.274825+0000", 2020-03-11T22:53:13.462 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.548576+0000" 2020-03-11T22:53:13.462 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.462 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.462 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.462 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.145417+0000", 2020-03-11T22:53:13.462 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.548595+0000" 2020-03-11T22:53:13.463 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.463 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.463 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.463 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.145398+0000", 2020-03-11T22:53:13.463 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.548601+0000" 2020-03-11T22:53:13.463 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.463 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.463 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.464 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.464 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "106", 2020-03-11T22:53:13.464 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.464 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.464 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.464 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.548601+0000", 2020-03-11T22:53:13.464 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.548824+0000" 2020-03-11T22:53:13.465 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.465 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.465 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.465 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.465 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "114", 2020-03-11T22:53:13.465 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.465 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.465 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.466 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.548830+0000", 2020-03-11T22:53:13.466 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.548844+0000" 2020-03-11T22:53:13.466 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.466 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.466 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.466 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.548853+0000", 2020-03-11T22:53:13.466 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.564845+0000" 2020-03-11T22:53:13.467 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.467 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.467 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.467 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.564846+0000", 2020-03-11T22:53:13.467 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.564919+0000" 2020-03-11T22:53:13.467 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.467 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.468 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:13.468 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.564920+0000", 2020-03-11T22:53:13.468 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.564938+0000" 2020-03-11T22:53:13.468 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.468 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.468 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:13.468 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.564939+0000", 2020-03-11T22:53:13.468 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.722844+0000" 2020-03-11T22:53:13.469 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.469 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.469 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.469 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.548848+0000", 2020-03-11T22:53:13.469 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:38.722855+0000" 2020-03-11T22:53:13.469 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.470 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.470 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:13.470 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.723075+0000", 2020-03-11T22:53:13.470 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:39.574129+0000" 2020-03-11T22:53:13.470 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.470 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.470 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:53:13.471 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:39.574130+0000", 2020-03-11T22:53:13.471 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:50.107557+0000" 2020-03-11T22:53:13.471 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.471 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.471 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:53:13.471 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:50.107557+0000", 2020-03-11T22:53:13.471 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:50.107951+0000" 2020-03-11T22:53:13.471 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.472 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.472 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:53:13.472 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:50.107952+0000", 2020-03-11T22:53:13.472 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:02.166010+0000" 2020-03-11T22:53:13.472 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.472 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.472 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/NotBackfilling", 2020-03-11T22:53:13.473 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:02.166010+0000", 2020-03-11T22:53:13.473 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:02.166173+0000" 2020-03-11T22:53:13.473 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.473 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.473 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:53:13.473 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:02.166173+0000", 2020-03-11T22:53:13.473 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:26.777897+0000" 2020-03-11T22:53:13.473 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.474 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.474 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:53:13.474 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:26.777898+0000", 2020-03-11T22:53:13.474 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:26.778424+0000" 2020-03-11T22:53:13.474 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.474 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.474 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:53:13.475 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:26.778424+0000", 2020-03-11T22:53:13.475 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:47.323856+0000" 2020-03-11T22:53:13.475 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.475 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.475 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:13.475 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:47.323857+0000", 2020-03-11T22:53:13.475 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:47.323996+0000" 2020-03-11T22:53:13.476 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.476 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.476 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.476 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:47.323996+0000", 2020-03-11T22:53:13.476 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:47.705326+0000" 2020-03-11T22:53:13.476 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.476 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.476 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:13.477 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.722855+0000", 2020-03-11T22:53:13.477 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:47.705364+0000" 2020-03-11T22:53:13.477 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.477 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.477 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.477 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.548844+0000", 2020-03-11T22:53:13.477 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:47.705395+0000" 2020-03-11T22:53:13.478 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.478 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.478 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.478 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:38.548824+0000", 2020-03-11T22:53:13.478 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:47.705405+0000" 2020-03-11T22:53:13.478 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.478 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.478 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.479 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.479 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "114", 2020-03-11T22:53:13.479 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.479 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.479 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.479 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:47.705406+0000", 2020-03-11T22:53:13.479 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:47.705838+0000" 2020-03-11T22:53:13.480 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.480 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.480 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.480 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.480 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.480 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.480 INFO:tasks.workunit.client.0.smithi155.stdout: "pg": "1.0", 2020-03-11T22:53:13.480 INFO:tasks.workunit.client.0.smithi155.stdout: "currently": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.481 INFO:tasks.workunit.client.0.smithi155.stdout: "history": [ 2020-03-11T22:53:13.481 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.481 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "26", 2020-03-11T22:53:13.481 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.481 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.481 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Initial", 2020-03-11T22:53:13.481 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:45.603080+0000", 2020-03-11T22:53:13.482 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:45.603191+0000" 2020-03-11T22:53:13.482 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.482 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.482 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.482 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.482 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "26", 2020-03-11T22:53:13.482 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.483 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.483 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.483 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:45.603192+0000", 2020-03-11T22:53:13.483 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:45.603228+0000" 2020-03-11T22:53:13.483 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.483 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.483 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.483 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.484 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "29", 2020-03-11T22:53:13.484 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.484 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.484 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.484 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:45.603232+0000", 2020-03-11T22:53:13.484 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:45.603243+0000" 2020-03-11T22:53:13.484 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.485 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.485 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.485 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:45.603252+0000", 2020-03-11T22:53:13.485 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:45.646434+0000" 2020-03-11T22:53:13.485 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.485 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.485 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.485 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:45.646435+0000", 2020-03-11T22:53:13.486 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:45.646509+0000" 2020-03-11T22:53:13.486 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.486 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.486 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:13.486 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:45.646510+0000", 2020-03-11T22:53:13.486 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:45.646526+0000" 2020-03-11T22:53:13.486 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.487 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.487 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:13.487 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:45.646526+0000", 2020-03-11T22:53:13.487 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:45.751565+0000" 2020-03-11T22:53:13.487 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.487 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.487 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.488 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:45.603248+0000", 2020-03-11T22:53:13.488 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:45.751575+0000" 2020-03-11T22:53:13.488 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.488 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.488 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:13.488 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:45.751774+0000", 2020-03-11T22:53:13.488 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.131653+0000" 2020-03-11T22:53:13.488 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.489 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.489 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:13.489 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.131654+0000", 2020-03-11T22:53:13.489 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.131743+0000" 2020-03-11T22:53:13.489 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.489 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.489 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.490 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.131744+0000", 2020-03-11T22:53:13.490 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.476369+0000" 2020-03-11T22:53:13.490 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.490 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.490 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:13.490 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:45.751576+0000", 2020-03-11T22:53:13.490 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.476388+0000" 2020-03-11T22:53:13.491 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.491 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.491 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.491 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:45.603244+0000", 2020-03-11T22:53:13.491 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.476402+0000" 2020-03-11T22:53:13.491 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.491 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.491 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.492 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:45.603228+0000", 2020-03-11T22:53:13.492 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.476407+0000" 2020-03-11T22:53:13.492 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.492 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.492 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.492 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.492 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "29", 2020-03-11T22:53:13.493 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.493 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.493 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.493 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.476408+0000", 2020-03-11T22:53:13.493 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.476626+0000" 2020-03-11T22:53:13.493 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.493 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.494 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.494 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.494 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "100", 2020-03-11T22:53:13.494 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.494 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.494 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.494 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.476631+0000", 2020-03-11T22:53:13.494 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.476643+0000" 2020-03-11T22:53:13.495 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.495 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.495 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.495 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.476652+0000", 2020-03-11T22:53:13.495 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.562252+0000" 2020-03-11T22:53:13.495 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.495 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.496 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.496 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.562253+0000", 2020-03-11T22:53:13.496 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.562322+0000" 2020-03-11T22:53:13.496 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.496 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.496 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:13.496 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.562323+0000", 2020-03-11T22:53:13.497 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.562335+0000" 2020-03-11T22:53:13.497 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.497 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.497 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:13.497 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.562336+0000", 2020-03-11T22:53:13.497 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.758163+0000" 2020-03-11T22:53:13.498 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.498 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.498 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.498 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.476648+0000", 2020-03-11T22:53:13.498 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.758173+0000" 2020-03-11T22:53:13.498 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.498 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.499 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:13.499 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.758324+0000", 2020-03-11T22:53:13.499 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.791917+0000" 2020-03-11T22:53:13.499 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.499 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.499 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:13.500 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.791918+0000", 2020-03-11T22:53:13.500 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:49:46.791966+0000" 2020-03-11T22:53:13.500 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.500 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.500 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.500 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.791967+0000", 2020-03-11T22:53:13.500 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.457204+0000" 2020-03-11T22:53:13.500 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.501 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.501 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:13.501 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.758174+0000", 2020-03-11T22:53:13.501 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.457246+0000" 2020-03-11T22:53:13.501 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.501 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.502 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.502 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.476644+0000", 2020-03-11T22:53:13.502 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.457319+0000" 2020-03-11T22:53:13.502 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.502 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.502 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.502 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:49:46.476626+0000", 2020-03-11T22:53:13.503 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.457328+0000" 2020-03-11T22:53:13.503 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.503 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.503 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.503 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.503 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "100", 2020-03-11T22:53:13.503 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.504 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.504 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.504 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.457328+0000", 2020-03-11T22:53:13.504 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.457752+0000" 2020-03-11T22:53:13.504 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.504 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.504 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.504 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.505 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "101", 2020-03-11T22:53:13.505 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.505 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.505 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.505 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.457760+0000", 2020-03-11T22:53:13.505 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.457783+0000" 2020-03-11T22:53:13.505 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.506 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.506 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.506 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.457800+0000", 2020-03-11T22:53:13.506 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.458621+0000" 2020-03-11T22:53:13.506 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.506 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.506 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.506 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.458622+0000", 2020-03-11T22:53:13.507 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.458733+0000" 2020-03-11T22:53:13.507 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.507 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.507 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.507 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.457792+0000", 2020-03-11T22:53:13.507 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.458752+0000" 2020-03-11T22:53:13.507 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.508 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.508 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/WaitActingChange", 2020-03-11T22:53:13.508 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.458753+0000", 2020-03-11T22:53:13.508 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.607492+0000" 2020-03-11T22:53:13.508 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.508 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.508 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.508 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.457784+0000", 2020-03-11T22:53:13.509 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.607511+0000" 2020-03-11T22:53:13.509 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.509 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.509 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.509 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.457753+0000", 2020-03-11T22:53:13.509 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.607518+0000" 2020-03-11T22:53:13.509 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.510 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.510 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.510 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.510 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "101", 2020-03-11T22:53:13.510 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.510 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.510 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.511 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.607518+0000", 2020-03-11T22:53:13.511 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.607770+0000" 2020-03-11T22:53:13.511 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.511 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.511 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.511 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.511 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "116", 2020-03-11T22:53:13.511 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.512 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.512 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Start", 2020-03-11T22:53:13.512 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.607776+0000", 2020-03-11T22:53:13.512 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.607791+0000" 2020-03-11T22:53:13.512 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.512 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.512 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetInfo", 2020-03-11T22:53:13.512 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.607801+0000", 2020-03-11T22:53:13.513 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.663689+0000" 2020-03-11T22:53:13.513 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.513 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.513 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetLog", 2020-03-11T22:53:13.513 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.663690+0000", 2020-03-11T22:53:13.513 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.663786+0000" 2020-03-11T22:53:13.513 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.514 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.514 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/GetMissing", 2020-03-11T22:53:13.514 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.663787+0000", 2020-03-11T22:53:13.514 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.663813+0000" 2020-03-11T22:53:13.514 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.514 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.514 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering/WaitUpThru", 2020-03-11T22:53:13.515 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.663813+0000", 2020-03-11T22:53:13.515 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.757765+0000" 2020-03-11T22:53:13.515 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.515 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.515 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Peering", 2020-03-11T22:53:13.515 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.607795+0000", 2020-03-11T22:53:13.515 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:32.757780+0000" 2020-03-11T22:53:13.515 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.516 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.516 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Activating", 2020-03-11T22:53:13.516 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.758091+0000", 2020-03-11T22:53:13.516 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:51:33.424109+0000" 2020-03-11T22:53:13.516 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.516 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.516 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/WaitLocalBackfillReserved", 2020-03-11T22:53:13.517 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:33.424110+0000", 2020-03-11T22:53:13.517 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:47.324157+0000" 2020-03-11T22:53:13.517 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.517 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.517 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/WaitRemoteBackfillReserved", 2020-03-11T22:53:13.517 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:47.324158+0000", 2020-03-11T22:53:13.517 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:52:47.324544+0000" 2020-03-11T22:53:13.517 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.518 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.518 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Backfilling", 2020-03-11T22:53:13.518 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:52:47.324544+0000", 2020-03-11T22:53:13.518 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:53:09.161233+0000" 2020-03-11T22:53:13.518 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.518 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.518 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Recovered", 2020-03-11T22:53:13.519 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:53:09.161234+0000", 2020-03-11T22:53:13.519 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:53:09.161375+0000" 2020-03-11T22:53:13.519 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.519 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.519 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active/Clean", 2020-03-11T22:53:13.519 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:53:09.161375+0000", 2020-03-11T22:53:13.519 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:53:09.344213+0000" 2020-03-11T22:53:13.520 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.520 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.520 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary/Active", 2020-03-11T22:53:13.520 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.757781+0000", 2020-03-11T22:53:13.520 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:53:09.344238+0000" 2020-03-11T22:53:13.520 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.520 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.521 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started/Primary", 2020-03-11T22:53:13.521 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.607792+0000", 2020-03-11T22:53:13.521 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:53:09.344258+0000" 2020-03-11T22:53:13.521 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.521 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.521 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Started", 2020-03-11T22:53:13.521 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:51:32.607771+0000", 2020-03-11T22:53:13.522 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:53:09.344265+0000" 2020-03-11T22:53:13.522 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.522 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.522 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:53:13.522 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.522 INFO:tasks.workunit.client.0.smithi155.stdout: "epoch": "116", 2020-03-11T22:53:13.522 INFO:tasks.workunit.client.0.smithi155.stdout: "states": [ 2020-03-11T22:53:13.523 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:53:13.523 INFO:tasks.workunit.client.0.smithi155.stdout: "state": "Reset", 2020-03-11T22:53:13.523 INFO:tasks.workunit.client.0.smithi155.stdout: "enter": "2020-03-11T22:53:09.344265+0000", 2020-03-11T22:53:13.523 INFO:tasks.workunit.client.0.smithi155.stdout: "exit": "2020-03-11T22:53:09.344562+0000" 2020-03-11T22:53:13.523 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.523 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.524 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.524 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.524 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:53:13.524 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:53:13.524 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:53:13.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:302: TEST_backfill_priority: '[' 0 '!=' 0 ']' 2020-03-11T22:53:13.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:306: TEST_backfill_priority: echo TEST PASSED 2020-03-11T22:53:13.525 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:309: TEST_backfill_priority: delete_pool test1 2020-03-11T22:53:13.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test1 2020-03-11T22:53:13.526 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test1 test1 --yes-i-really-really-mean-it 2020-03-11T22:53:13.526 INFO:tasks.workunit.client.0.smithi155.stdout:TEST PASSED 2020-03-11T22:53:13.834 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test1' does not exist 2020-03-11T22:53:13.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:310: TEST_backfill_priority: delete_pool test3 2020-03-11T22:53:13.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test3 2020-03-11T22:53:13.842 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test3 test3 --yes-i-really-really-mean-it 2020-03-11T22:53:14.347 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test3' does not exist 2020-03-11T22:53:14.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:311: TEST_backfill_priority: delete_pool test7 2020-03-11T22:53:14.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: delete_pool: local poolname=test7 2020-03-11T22:53:14.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: delete_pool: ceph osd pool delete test7 test7 --yes-i-really-really-mean-it 2020-03-11T22:53:14.873 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test7' does not exist 2020-03-11T22:53:14.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:312: TEST_backfill_priority: kill_daemons td/osd-backfill-prio 2020-03-11T22:53:14.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:14.882 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:14.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:14.882 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:14.883 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:14.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:14.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:313: TEST_backfill_priority: return 0 2020-03-11T22:53:14.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-prio.sh:40: run: teardown td/osd-backfill-prio 2020-03-11T22:53:14.991 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:53:14.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:53:14.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:53:14.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:14.992 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:14.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:14.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:14.992 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:14.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:14.997 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:53:14.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:53:14.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:53:14.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:53:14.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:53:14.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:53:15.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:15.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:53:15.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:53:15.001 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:15.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:53:15.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:53:15.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:53:15.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:53:15.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:15.029 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:53:15.029 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56110 2020-03-11T22:53:15.030 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:53:15.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:53:15.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2146: main: code=0 2020-03-11T22:53:15.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: main: teardown td/osd-backfill-prio 0 2020-03-11T22:53:15.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-prio 2020-03-11T22:53:15.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs=0 2020-03-11T22:53:15.031 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-prio KILL 2020-03-11T22:53:15.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:15.032 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:15.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:15.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:15.032 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:15.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:15.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:53:15.033 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:53:15.033 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:53:15.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:53:15.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:53:15.034 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:53:15.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:15.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:53:15.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:53:15.037 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:15.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:53:15.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o 0 = 1 ']' 2020-03-11T22:53:15.038 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-prio 2020-03-11T22:53:15.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:53:15.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:15.039 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.56110 2020-03-11T22:53:15.039 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.56110 2020-03-11T22:53:15.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:53:15.040 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:53:15.041 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: main: return 0 2020-03-11T22:53:15.041 INFO:teuthology.orchestra.run:Running command with timeout 3600 2020-03-11T22:53:15.041 INFO:teuthology.orchestra.run.smithi155:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2020-03-11T22:53:15.116 INFO:tasks.workunit:Running workunit osd/osd-backfill-recovery-log.sh... 2020-03-11T22:53:15.116 INFO:teuthology.orchestra.run.smithi155:workunit test osd/osd-backfill-recovery-log.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=4acc1121e47039a3cdf2978ae458c10d153aa141 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh 2020-03-11T22:53:15.174 INFO:tasks.workunit.client.0.smithi155.stderr:stty: 'standard input': Inappropriate ioctl for device 2020-03-11T22:53:15.182 INFO:tasks.workunit.client.0.smithi155.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2020-03-11T22:53:15.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: export PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:53:15.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2139: main: PATH=.:/home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin 2020-03-11T22:53:15.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: export PYTHONWARNINGS=ignore 2020-03-11T22:53:15.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2140: main: PYTHONWARNINGS=ignore 2020-03-11T22:53:15.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: export CEPH_CONF=/dev/null 2020-03-11T22:53:15.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2141: main: CEPH_CONF=/dev/null 2020-03-11T22:53:15.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2142: main: unset CEPH_ARGS 2020-03-11T22:53:15.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2144: main: local code 2020-03-11T22:53:15.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2145: main: run td/osd-backfill-recovery-log 2020-03-11T22:53:15.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:21: run: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:15.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:22: run: shift 2020-03-11T22:53:15.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: export CEPH_MON=127.0.0.1:7129 2020-03-11T22:53:15.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:25: run: CEPH_MON=127.0.0.1:7129 2020-03-11T22:53:15.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:26: run: export CEPH_ARGS 2020-03-11T22:53:15.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: uuidgen 2020-03-11T22:53:15.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:27: run: CEPH_ARGS+='--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none ' 2020-03-11T22:53:15.184 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:28: run: CEPH_ARGS+='--mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:15.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: set 2020-03-11T22:53:15.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2020-03-11T22:53:15.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:30: run: local 'funcs=TEST_backfill_log_1 2020-03-11T22:53:15.187 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_backfill_log_2 2020-03-11T22:53:15.187 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_recovery_1 2020-03-11T22:53:15.187 INFO:tasks.workunit.client.0.smithi155.stderr:TEST_recovery_2' 2020-03-11T22:53:15.187 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:53:15.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:53:15.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:15.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:53:15.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:15.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:53:15.188 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:53:15.188 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:53:15.188 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:53:15.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:53:15.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:53:15.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:53:15.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:53:15.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:53:15.189 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:53:15.189 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:53:15.190 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:53:15.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:53:15.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:53:15.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:15.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:53:15.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:53:15.191 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:53:15.191 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:53:15.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:53:15.192 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:53:15.192 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:53:15.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:15.193 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:53:15.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76480 2020-03-11T22:53:15.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:53:15.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:53:15.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:53:15.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:53:15.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:15.194 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:53:15.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76480 2020-03-11T22:53:15.195 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:53:15.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 1024 -le 1024 ']' 2020-03-11T22:53:15.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:136: setup: ulimit -n 4096 2020-03-11T22:53:15.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:53:15.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:53:15.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_1 td/osd-backfill-recovery-log 2020-03-11T22:53:15.197 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:103: TEST_backfill_log_1: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:15.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:105: TEST_backfill_log_1: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2 8 150 2020-03-11T22:53:15.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:15.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:15.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:53:15.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=8 2020-03-11T22:53:15.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:53:15.198 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:53:15.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:53:15.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:53:15.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:15.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:53:15.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:53:15.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:53:15.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:53:15.199 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:53:15.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:53:15.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:15.529 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:15.530 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:15.530 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:15.530 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:53:15.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:15.530 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:53:15.714 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:53:15.714 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:53:15.714 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:53:15.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:53:15.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:53:15.715 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:53:15.715 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:53:15.715 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:53:15.715 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:53:15.715 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:15.715 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:15.716 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:53:15.716 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76480/ceph-mon.a.asok 2020-03-11T22:53:15.716 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:53:15.716 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76480/ceph-mon.a.asok config get fsid 2020-03-11T22:53:15.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:53:15.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:53:15.801 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:53:15.802 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:53:15.802 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:53:15.802 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:53:15.802 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:53:15.802 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:53:15.802 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:53:15.802 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:15.803 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:53:15.803 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76480/ceph-mon.a.asok 2020-03-11T22:53:15.803 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:53:15.803 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76480/ceph-mon.a.asok config get mon_host 2020-03-11T22:53:15.887 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:53:15.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:15.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:53:15.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:53:15.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:53:15.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:53:15.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:53:16.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:53:16.763 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:16.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:16.764 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:16.764 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:16.764 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:53:16.764 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:16.765 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:53:16.765 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:53:16.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:53:16.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:16.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:16.786 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:53:16.786 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:53:16.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:16.790 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:53:16.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:16.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:16.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:53:16.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:16.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:53:16.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:16.791 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:16.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:16.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:16.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:53:16.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:53:16.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:16.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:16.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:16.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:16.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:16.793 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:16.793 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:16.793 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:16.794 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:53:16.794 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:16.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:16.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:16.794 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:16.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:16.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:16.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:16.795 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:16.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:16.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:16.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:16.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:16.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:53:16.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:16.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=0ebaf6fd-372d-421a-b33f-630f51df43e1 2020-03-11T22:53:16.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 0ebaf6fd-372d-421a-b33f-630f51df43e1' 2020-03-11T22:53:16.797 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:16.797 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 0ebaf6fd-372d-421a-b33f-630f51df43e1 2020-03-11T22:53:16.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDca2le8X7ELxAAYxgo/LPlGUSuU+ku+TRclA== 2020-03-11T22:53:16.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDca2le8X7ELxAAYxgo/LPlGUSuU+ku+TRclA=="}' 2020-03-11T22:53:16.804 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 0ebaf6fd-372d-421a-b33f-630f51df43e1 -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:53:17.137 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:53:17.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:53:17.147 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDca2le8X7ELxAAYxgo/LPlGUSuU+ku+TRclA== --osd-uuid 0ebaf6fd-372d-421a-b33f-630f51df43e1 2020-03-11T22:53:17.171 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:17.167+0000 7f5c456f1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:17.173 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:17.169+0000 7f5c456f1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:17.174 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:17.171+0000 7f5c456f1ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:17.175 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:17.172+0000 7f5c456f1ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:53:17.175 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:17.172+0000 7f5c456f1ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:53:17.176 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:17.172+0000 7f5c456f1ec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:53:17.213 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:17.209+0000 7f5c456f1ec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:53:20.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:53:20.233 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:20.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:53:20.234 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:20.234 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2020-03-11T22:53:20.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:53:20.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:20.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:20.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:20.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:20.681 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:53:20.701 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:20.698+0000 7fbf87c67ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:20.703 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:20.700+0000 7fbf87c67ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:20.705 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:20.701+0000 7fbf87c67ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:21.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:53:21.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:21.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:53:21.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:21.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:21.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:21.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:21.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:21.028 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:53:21.028 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:53:21.221 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:21.217+0000 7fbf87c67ec0 -1 Falling back to public interface 2020-03-11T22:53:21.372 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:22.004 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:22.000+0000 7fbf87c67ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:53:22.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:22.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:22.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:22.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:22.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:53:22.375 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:53:22.745 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/76912,v1:127.0.0.1:6803/76912] [v2:127.0.0.1:6804/76912,v1:127.0.0.1:6805/76912] exists,up 0ebaf6fd-372d-421a-b33f-630f51df43e1 2020-03-11T22:53:22.745 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 fe091d0e-6db1-4028-8b60-a5a49d909f79 2020-03-11T22:53:22.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:22.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:22.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:22.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:22.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:53:22.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:22.747 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:22.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:53:22.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:22.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:53:22.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:22.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:22.748 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:22.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:22.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:53:22.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:53:22.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:22.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:22.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:22.749 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:22.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:22.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:22.750 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:22.750 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:22.750 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:53:22.750 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:22.750 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:22.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:22.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:22.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:22.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:22.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:22.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:22.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:22.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:22.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:22.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:22.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:53:22.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:22.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=fe091d0e-6db1-4028-8b60-a5a49d909f79 2020-03-11T22:53:22.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 fe091d0e-6db1-4028-8b60-a5a49d909f79' 2020-03-11T22:53:22.752 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:22.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDia2le+EYPLRAAZ64n5B3M10tIAeHLGkDkWQ== 2020-03-11T22:53:22.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDia2le+EYPLRAAZ64n5B3M10tIAeHLGkDkWQ=="}' 2020-03-11T22:53:22.759 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new fe091d0e-6db1-4028-8b60-a5a49d909f79 -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:53:23.232 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:53:23.242 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:53:23.243 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDia2le+EYPLRAAZ64n5B3M10tIAeHLGkDkWQ== --osd-uuid fe091d0e-6db1-4028-8b60-a5a49d909f79 2020-03-11T22:53:23.267 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:23.263+0000 7f6f70de5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:23.268 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:23.265+0000 7f6f70de5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:23.270 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:23.267+0000 7f6f70de5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:23.271 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:23.267+0000 7f6f70de5ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:53:23.271 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:23.267+0000 7f6f70de5ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:53:23.271 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:23.267+0000 7f6f70de5ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:53:23.344 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:23.341+0000 7f6f70de5ec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:53:26.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:53:26.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:26.426 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2020-03-11T22:53:26.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:53:26.426 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:26.880 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:53:26.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:53:26.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:26.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:26.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:26.881 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:26.902 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:26.898+0000 7f8e281f2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:26.904 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:26.900+0000 7f8e281f2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:26.906 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:26.902+0000 7f8e281f2ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:27.251 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:53:27.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:27.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:53:27.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:27.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:27.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:27.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:27.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:27.253 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:53:27.253 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:53:27.429 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:27.425+0000 7f8e281f2ec0 -1 Falling back to public interface 2020-03-11T22:53:27.592 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:28.215 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:28.211+0000 7f8e281f2ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:53:28.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:28.593 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:28.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:28.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:28.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:53:28.594 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:53:28.996 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/77275,v1:127.0.0.1:6811/77275] [v2:127.0.0.1:6812/77275,v1:127.0.0.1:6813/77275] exists,up fe091d0e-6db1-4028-8b60-a5a49d909f79 2020-03-11T22:53:28.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:28.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:28.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:28.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:28.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:53:28.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:28.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:28.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:53:28.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:28.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:53:28.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:28.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:28.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:28.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:28.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:53:28.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:53:28.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:28.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:28.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:28.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:29.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:29.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:29.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:29.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:29.000 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:53:29.000 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:29.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:29.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:29.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:29.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:29.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:29.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:53:29.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:29.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d7008f99-e778-43f7-a34f-3d7a6d59553a 2020-03-11T22:53:29.003 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 d7008f99-e778-43f7-a34f-3d7a6d59553a' 2020-03-11T22:53:29.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:29.003 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 d7008f99-e778-43f7-a34f-3d7a6d59553a 2020-03-11T22:53:29.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDpa2le/NqNABAASoqt8YyjW6G3R4tRcQJi/g== 2020-03-11T22:53:29.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDpa2le/NqNABAASoqt8YyjW6G3R4tRcQJi/g=="}' 2020-03-11T22:53:29.012 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d7008f99-e778-43f7-a34f-3d7a6d59553a -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:53:29.448 INFO:tasks.workunit.client.0.smithi155.stdout:2 2020-03-11T22:53:29.457 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:53:29.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDpa2le/NqNABAASoqt8YyjW6G3R4tRcQJi/g== --osd-uuid d7008f99-e778-43f7-a34f-3d7a6d59553a 2020-03-11T22:53:29.482 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:29.478+0000 7fa624345ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:29.483 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:29.480+0000 7fa624345ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:29.485 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:29.482+0000 7fa624345ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:29.486 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:29.483+0000 7fa624345ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:53:29.486 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:29.483+0000 7fa624345ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:53:29.486 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:29.483+0000 7fa624345ec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:53:29.592 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:29.589+0000 7fa624345ec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:53:32.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:53:32.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:32.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:53:32.994 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:32.995 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2020-03-11T22:53:33.452 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:53:33.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:53:33.452 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:33.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:33.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:33.453 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:33.473 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:33.469+0000 7f023ee9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:33.475 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:33.471+0000 7f023ee9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:33.477 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:33.472+0000 7f023ee9aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:33.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:53:33.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:33.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:53:33.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:33.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:33.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:33.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:33.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:33.797 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:53:33.797 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:53:33.992 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:33.988+0000 7f023ee9aec0 -1 Falling back to public interface 2020-03-11T22:53:34.154 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:34.759 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:34.756+0000 7f023ee9aec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:53:35.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:35.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:35.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:35.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:35.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:53:35.157 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:53:35.545 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/77630,v1:127.0.0.1:6819/77630] [v2:127.0.0.1:6820/77630,v1:127.0.0.1:6821/77630] exists,up d7008f99-e778-43f7-a34f-3d7a6d59553a 2020-03-11T22:53:35.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:35.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:35.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:35.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:35.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:53:35.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:35.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:35.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:53:35.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:35.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:53:35.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:35.547 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:35.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:35.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:35.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:53:35.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:53:35.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:35.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:35.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:35.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:35.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:35.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:35.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:35.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:35.549 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:53:35.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:35.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:35.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:35.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:35.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:35.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:35.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:35.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:35.550 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:35.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:35.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:35.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:35.551 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:53:35.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:35.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e9f1eed4-db83-4540-804c-022c287b8984 2020-03-11T22:53:35.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 e9f1eed4-db83-4540-804c-022c287b8984' 2020-03-11T22:53:35.552 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:35.553 INFO:tasks.workunit.client.0.smithi155.stdout:add osd3 e9f1eed4-db83-4540-804c-022c287b8984 2020-03-11T22:53:35.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQDva2leocpFIRAALzRN9c4QJBtPkDk7eY9Tjg== 2020-03-11T22:53:35.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQDva2leocpFIRAALzRN9c4QJBtPkDk7eY9Tjg=="}' 2020-03-11T22:53:35.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e9f1eed4-db83-4540-804c-022c287b8984 -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:53:36.007 INFO:tasks.workunit.client.0.smithi155.stdout:3 2020-03-11T22:53:36.017 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:53:36.018 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQDva2leocpFIRAALzRN9c4QJBtPkDk7eY9Tjg== --osd-uuid e9f1eed4-db83-4540-804c-022c287b8984 2020-03-11T22:53:36.041 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:36.039+0000 7f0efd142ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:36.043 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:36.040+0000 7f0efd142ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:36.044 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:36.041+0000 7f0efd142ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:36.045 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:36.042+0000 7f0efd142ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:53:36.045 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:36.042+0000 7f0efd142ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:53:36.045 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:36.042+0000 7f0efd142ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:53:36.092 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:36.089+0000 7f0efd142ec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:53:39.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:53:39.249 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:39.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:53:39.250 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:39.250 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd3 key to auth repository 2020-03-11T22:53:39.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:53:39.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:39.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:39.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:39.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:39.709 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.3 2020-03-11T22:53:39.729 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:39.727+0000 7f89c4c20ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:39.731 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:39.729+0000 7f89c4c20ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:39.734 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:39.731+0000 7f89c4c20ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:40.051 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:53:40.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:40.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:53:40.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:40.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:40.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:40.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:40.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:40.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:53:40.053 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:53:40.249 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:40.246+0000 7f89c4c20ec0 -1 Falling back to public interface 2020-03-11T22:53:40.398 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:41.034 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:41.032+0000 7f89c4c20ec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:53:41.399 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:41.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:41.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:41.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:41.400 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:53:41.400 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:53:41.825 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:42.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:42.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:42.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:53:42.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:42.827 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:53:42.827 INFO:tasks.workunit.client.0.smithi155.stdout:2 2020-03-11T22:53:43.176 INFO:tasks.workunit.client.0.smithi155.stdout:osd.3 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/77985,v1:127.0.0.1:6827/77985] [v2:127.0.0.1:6828/77985,v1:127.0.0.1:6829/77985] exists,up e9f1eed4-db83-4540-804c-022c287b8984 2020-03-11T22:53:43.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:43.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:43.176 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:43.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:43.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-03-11T22:53:43.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:43.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:43.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:53:43.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:43.177 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:53:43.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:43.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:43.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:43.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:43.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-03-11T22:53:43.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-03-11T22:53:43.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:43.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:43.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:43.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:43.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:43.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:43.179 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:43.179 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:43.179 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:53:43.180 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:43.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:43.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:43.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:43.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:43.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:43.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:43.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:43.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:43.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:43.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:43.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:43.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-03-11T22:53:43.183 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:43.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e7f4412b-0c99-4e2a-9af1-851032a43cf1 2020-03-11T22:53:43.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 e7f4412b-0c99-4e2a-9af1-851032a43cf1' 2020-03-11T22:53:43.184 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:43.184 INFO:tasks.workunit.client.0.smithi155.stdout:add osd4 e7f4412b-0c99-4e2a-9af1-851032a43cf1 2020-03-11T22:53:43.193 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD3a2leKEZaCxAA+Gc4ERk7OxxL+qzjufQDxw== 2020-03-11T22:53:43.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD3a2leKEZaCxAA+Gc4ERk7OxxL+qzjufQDxw=="}' 2020-03-11T22:53:43.194 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e7f4412b-0c99-4e2a-9af1-851032a43cf1 -i td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:53:43.665 INFO:tasks.workunit.client.0.smithi155.stdout:4 2020-03-11T22:53:43.674 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:53:43.675 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD3a2leKEZaCxAA+Gc4ERk7OxxL+qzjufQDxw== --osd-uuid e7f4412b-0c99-4e2a-9af1-851032a43cf1 2020-03-11T22:53:43.699 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:43.696+0000 7fd011006ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:43.700 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:43.698+0000 7fd011006ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:43.701 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:43.699+0000 7fd011006ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:43.702 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:43.699+0000 7fd011006ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:53:43.702 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:43.699+0000 7fd011006ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:53:43.702 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:43.699+0000 7fd011006ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:53:43.751 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:43.749+0000 7fd011006ec0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-03-11T22:53:47.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-03-11T22:53:47.310 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:47.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:53:47.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:47.311 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd4 key to auth repository 2020-03-11T22:53:47.767 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.4 2020-03-11T22:53:47.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:53:47.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:47.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:47.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:47.768 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:47.789 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:47.787+0000 7fad9d72eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:47.791 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:47.788+0000 7fad9d72eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:47.792 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:47.789+0000 7fad9d72eec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:48.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:53:48.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:48.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:53:48.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:48.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:48.116 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:48.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:48.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:48.117 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:53:48.117 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:53:48.312 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:48.309+0000 7fad9d72eec0 -1 Falling back to public interface 2020-03-11T22:53:48.461 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:49.053 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:49.050+0000 7fad9d72eec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:53:49.462 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:49.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:49.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:49.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:49.463 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:53:49.464 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:53:49.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:50.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:50.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:50.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 2 2020-03-11T22:53:50.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:50.814 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:53:50.815 INFO:tasks.workunit.client.0.smithi155.stdout:2 2020-03-11T22:53:51.167 INFO:tasks.workunit.client.0.smithi155.stdout:osd.4 up in weight 1 up_from 22 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/78368,v1:127.0.0.1:6835/78368] [v2:127.0.0.1:6836/78368,v1:127.0.0.1:6837/78368] exists,up e7f4412b-0c99-4e2a-9af1-851032a43cf1 2020-03-11T22:53:51.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:51.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:51.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:51.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:53:51.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-03-11T22:53:51.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:53:51.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:53:51.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-03-11T22:53:51.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:53:51.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-03-11T22:53:51.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:53:51.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:53:51.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:53:51.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:53:51.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-03-11T22:53:51.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-03-11T22:53:51.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:53:51.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10' 2020-03-11T22:53:51.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:53:51.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:53:51.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:53:51.170 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:53:51.170 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:53:51.170 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:53:51.171 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:53:51.171 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:51.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:53:51.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:53:51.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:53:51.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:53:51.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:53:51.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:53:51.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:53:51.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:53:51.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:51.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:53:51.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:53:51.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-03-11T22:53:51.173 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:53:51.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=b3af1cc6-fd4c-405d-b2d5-86c8a703d116 2020-03-11T22:53:51.173 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 b3af1cc6-fd4c-405d-b2d5-86c8a703d116' 2020-03-11T22:53:51.174 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:53:51.174 INFO:tasks.workunit.client.0.smithi155.stdout:add osd5 b3af1cc6-fd4c-405d-b2d5-86c8a703d116 2020-03-11T22:53:51.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQD/a2le0S27ChAAtBpeNaWQwG/dOelOh3hTMw== 2020-03-11T22:53:51.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQD/a2le0S27ChAAtBpeNaWQwG/dOelOh3hTMw=="}' 2020-03-11T22:53:51.183 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new b3af1cc6-fd4c-405d-b2d5-86c8a703d116 -i td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:53:51.726 INFO:tasks.workunit.client.0.smithi155.stdout:5 2020-03-11T22:53:51.735 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:53:51.736 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQD/a2le0S27ChAAtBpeNaWQwG/dOelOh3hTMw== --osd-uuid b3af1cc6-fd4c-405d-b2d5-86c8a703d116 2020-03-11T22:53:51.761 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:51.758+0000 7ff94d5a5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:51.762 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:51.760+0000 7ff94d5a5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:51.764 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:51.761+0000 7ff94d5a5ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:51.764 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:51.762+0000 7ff94d5a5ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:53:51.765 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:51.762+0000 7ff94d5a5ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:53:51.765 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:51.762+0000 7ff94d5a5ec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:53:51.865 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:51.863+0000 7ff94d5a5ec0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-03-11T22:53:55.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/5/keyring 2020-03-11T22:53:55.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:53:55.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd5 key to auth repository 2020-03-11T22:53:55.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:53:55.266 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd5 key to auth repository 2020-03-11T22:53:55.722 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.5 2020-03-11T22:53:55.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:53:55.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 5 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:53:55.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:53:55.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:53:55.724 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.5 2020-03-11T22:53:55.744 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:55.741+0000 7f18c02c6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:55.746 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:55.743+0000 7f18c02c6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:55.747 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:55.744+0000 7f18c02c6ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:53:56.068 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 5 2020-03-11T22:53:56.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:53:56.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=5 2020-03-11T22:53:56.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:53:56.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:53:56.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:56.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:53:56.069 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:56.070 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:53:56.070 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:53:56.264 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:56.261+0000 7f18c02c6ec0 -1 Falling back to public interface 2020-03-11T22:53:56.415 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:53:57.056 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:53:57.052+0000 7f18c02c6ec0 -1 osd.5 0 log_to_monitors {default=true} 2020-03-11T22:53:57.416 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:53:57.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:53:57.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:53:57.417 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:53:57.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.5 up' 2020-03-11T22:53:57.418 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:53:57.795 INFO:tasks.workunit.client.0.smithi155.stdout:osd.5 up in weight 1 up_from 27 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/78751,v1:127.0.0.1:6843/78751] [v2:127.0.0.1:6844/78751,v1:127.0.0.1:6845/78751] exists,up b3af1cc6-fd4c-405d-b2d5-86c8a703d116 2020-03-11T22:53:57.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:53:57.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:53:57.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:53:57.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:59: _common_test: create_pool test 1 1 2020-03-11T22:53:57.796 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:539: create_pool: ceph osd pool create test 1 1 2020-03-11T22:53:58.474 INFO:tasks.workunit.client.0.smithi155.stderr:pool 'test' already exists 2020-03-11T22:53:58.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:540: create_pool: sleep 1 2020-03-11T22:53:59.484 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: seq 1 150 2020-03-11T22:53:59.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:59.485 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-1 /etc/passwd 2020-03-11T22:53:59.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:59.852 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-2 /etc/passwd 2020-03-11T22:53:59.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:53:59.929 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-3 /etc/passwd 2020-03-11T22:54:00.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:00.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-4 /etc/passwd 2020-03-11T22:54:00.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:00.130 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-5 /etc/passwd 2020-03-11T22:54:00.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:00.303 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-6 /etc/passwd 2020-03-11T22:54:00.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:00.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-7 /etc/passwd 2020-03-11T22:54:00.489 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:00.490 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-8 /etc/passwd 2020-03-11T22:54:00.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:00.538 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-9 /etc/passwd 2020-03-11T22:54:00.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:00.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-10 /etc/passwd 2020-03-11T22:54:00.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:00.671 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-11 /etc/passwd 2020-03-11T22:54:00.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:00.746 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-12 /etc/passwd 2020-03-11T22:54:00.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:00.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-13 /etc/passwd 2020-03-11T22:54:00.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:00.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-14 /etc/passwd 2020-03-11T22:54:00.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:00.971 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-15 /etc/passwd 2020-03-11T22:54:01.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:01.046 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-16 /etc/passwd 2020-03-11T22:54:01.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:01.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-17 /etc/passwd 2020-03-11T22:54:01.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:01.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-18 /etc/passwd 2020-03-11T22:54:01.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:01.374 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-19 /etc/passwd 2020-03-11T22:54:01.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:01.473 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-20 /etc/passwd 2020-03-11T22:54:01.615 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:01.616 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-21 /etc/passwd 2020-03-11T22:54:01.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:01.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-22 /etc/passwd 2020-03-11T22:54:01.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:01.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-23 /etc/passwd 2020-03-11T22:54:01.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:01.867 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-24 /etc/passwd 2020-03-11T22:54:01.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:01.933 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-25 /etc/passwd 2020-03-11T22:54:01.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:01.989 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-26 /etc/passwd 2020-03-11T22:54:02.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:02.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-27 /etc/passwd 2020-03-11T22:54:02.225 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:02.226 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-28 /etc/passwd 2020-03-11T22:54:02.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:02.280 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-29 /etc/passwd 2020-03-11T22:54:02.363 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:02.364 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-30 /etc/passwd 2020-03-11T22:54:02.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:02.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-31 /etc/passwd 2020-03-11T22:54:02.513 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:02.514 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-32 /etc/passwd 2020-03-11T22:54:02.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:02.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-33 /etc/passwd 2020-03-11T22:54:02.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:02.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-34 /etc/passwd 2020-03-11T22:54:02.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:02.785 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-35 /etc/passwd 2020-03-11T22:54:02.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:02.839 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-36 /etc/passwd 2020-03-11T22:54:02.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:02.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-37 /etc/passwd 2020-03-11T22:54:03.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.019 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-38 /etc/passwd 2020-03-11T22:54:03.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.072 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-39 /etc/passwd 2020-03-11T22:54:03.122 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.123 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-40 /etc/passwd 2020-03-11T22:54:03.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-41 /etc/passwd 2020-03-11T22:54:03.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.345 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-42 /etc/passwd 2020-03-11T22:54:03.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.397 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-43 /etc/passwd 2020-03-11T22:54:03.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.447 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-44 /etc/passwd 2020-03-11T22:54:03.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.528 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-45 /etc/passwd 2020-03-11T22:54:03.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-46 /etc/passwd 2020-03-11T22:54:03.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-47 /etc/passwd 2020-03-11T22:54:03.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.754 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-48 /etc/passwd 2020-03-11T22:54:03.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.838 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-49 /etc/passwd 2020-03-11T22:54:03.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-50 /etc/passwd 2020-03-11T22:54:03.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:03.964 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-51 /etc/passwd 2020-03-11T22:54:04.055 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:04.056 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-52 /etc/passwd 2020-03-11T22:54:04.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:04.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-53 /etc/passwd 2020-03-11T22:54:04.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:04.206 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-54 /etc/passwd 2020-03-11T22:54:04.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:04.356 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-55 /etc/passwd 2020-03-11T22:54:04.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:04.448 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-56 /etc/passwd 2020-03-11T22:54:04.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:04.498 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-57 /etc/passwd 2020-03-11T22:54:04.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:04.581 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-58 /etc/passwd 2020-03-11T22:54:04.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:04.673 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-59 /etc/passwd 2020-03-11T22:54:04.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:04.723 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-60 /etc/passwd 2020-03-11T22:54:04.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:04.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-61 /etc/passwd 2020-03-11T22:54:04.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:04.932 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-62 /etc/passwd 2020-03-11T22:54:05.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.024 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-63 /etc/passwd 2020-03-11T22:54:05.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.073 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-64 /etc/passwd 2020-03-11T22:54:05.131 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.132 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-65 /etc/passwd 2020-03-11T22:54:05.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-66 /etc/passwd 2020-03-11T22:54:05.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-67 /etc/passwd 2020-03-11T22:54:05.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-68 /etc/passwd 2020-03-11T22:54:05.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-69 /etc/passwd 2020-03-11T22:54:05.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-70 /etc/passwd 2020-03-11T22:54:05.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.632 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-71 /etc/passwd 2020-03-11T22:54:05.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-72 /etc/passwd 2020-03-11T22:54:05.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.776 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-73 /etc/passwd 2020-03-11T22:54:05.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.823 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-74 /etc/passwd 2020-03-11T22:54:05.898 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.899 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-75 /etc/passwd 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:05.993 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-76 /etc/passwd 2020-03-11T22:54:06.088 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:06.089 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-77 /etc/passwd 2020-03-11T22:54:06.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:06.148 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-78 /etc/passwd 2020-03-11T22:54:06.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:06.265 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-79 /etc/passwd 2020-03-11T22:54:06.369 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:06.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-80 /etc/passwd 2020-03-11T22:54:06.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:06.424 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-81 /etc/passwd 2020-03-11T22:54:06.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:06.474 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-82 /etc/passwd 2020-03-11T22:54:06.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:06.586 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-83 /etc/passwd 2020-03-11T22:54:06.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:06.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-84 /etc/passwd 2020-03-11T22:54:06.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:06.749 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-85 /etc/passwd 2020-03-11T22:54:06.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:06.799 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-86 /etc/passwd 2020-03-11T22:54:06.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:06.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-87 /etc/passwd 2020-03-11T22:54:06.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:06.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-88 /etc/passwd 2020-03-11T22:54:07.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:07.049 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-89 /etc/passwd 2020-03-11T22:54:07.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:07.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-90 /etc/passwd 2020-03-11T22:54:07.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:07.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-91 /etc/passwd 2020-03-11T22:54:07.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:07.441 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-92 /etc/passwd 2020-03-11T22:54:07.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:07.491 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-93 /etc/passwd 2020-03-11T22:54:07.610 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:07.610 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-94 /etc/passwd 2020-03-11T22:54:07.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:07.666 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-95 /etc/passwd 2020-03-11T22:54:07.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:07.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-96 /etc/passwd 2020-03-11T22:54:07.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:07.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-97 /etc/passwd 2020-03-11T22:54:07.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:07.949 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-98 /etc/passwd 2020-03-11T22:54:07.999 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.000 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-99 /etc/passwd 2020-03-11T22:54:08.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.050 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-100 /etc/passwd 2020-03-11T22:54:08.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.100 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-101 /etc/passwd 2020-03-11T22:54:08.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.266 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-102 /etc/passwd 2020-03-11T22:54:08.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-103 /etc/passwd 2020-03-11T22:54:08.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.425 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-104 /etc/passwd 2020-03-11T22:54:08.533 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-105 /etc/passwd 2020-03-11T22:54:08.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-106 /etc/passwd 2020-03-11T22:54:08.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-107 /etc/passwd 2020-03-11T22:54:08.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.726 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-108 /etc/passwd 2020-03-11T22:54:08.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-109 /etc/passwd 2020-03-11T22:54:08.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-110 /etc/passwd 2020-03-11T22:54:08.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:08.951 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-111 /etc/passwd 2020-03-11T22:54:09.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-112 /etc/passwd 2020-03-11T22:54:09.108 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.109 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-113 /etc/passwd 2020-03-11T22:54:09.158 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.159 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-114 /etc/passwd 2020-03-11T22:54:09.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.252 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-115 /etc/passwd 2020-03-11T22:54:09.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.300 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-116 /etc/passwd 2020-03-11T22:54:09.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.350 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-117 /etc/passwd 2020-03-11T22:54:09.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-118 /etc/passwd 2020-03-11T22:54:09.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.561 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-119 /etc/passwd 2020-03-11T22:54:09.608 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.609 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-120 /etc/passwd 2020-03-11T22:54:09.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-121 /etc/passwd 2020-03-11T22:54:09.786 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.787 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-122 /etc/passwd 2020-03-11T22:54:09.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.834 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-123 /etc/passwd 2020-03-11T22:54:09.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.884 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-124 /etc/passwd 2020-03-11T22:54:09.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:09.987 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-125 /etc/passwd 2020-03-11T22:54:10.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:10.096 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-126 /etc/passwd 2020-03-11T22:54:10.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:10.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-127 /etc/passwd 2020-03-11T22:54:10.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:10.259 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-128 /etc/passwd 2020-03-11T22:54:10.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:10.430 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-129 /etc/passwd 2020-03-11T22:54:10.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:10.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-130 /etc/passwd 2020-03-11T22:54:10.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:10.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-131 /etc/passwd 2020-03-11T22:54:10.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:10.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-132 /etc/passwd 2020-03-11T22:54:10.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:10.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-133 /etc/passwd 2020-03-11T22:54:10.792 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:10.793 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-134 /etc/passwd 2020-03-11T22:54:10.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:10.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-135 /etc/passwd 2020-03-11T22:54:10.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:10.940 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-136 /etc/passwd 2020-03-11T22:54:11.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.009 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-137 /etc/passwd 2020-03-11T22:54:11.059 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.060 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-138 /etc/passwd 2020-03-11T22:54:11.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.182 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-139 /etc/passwd 2020-03-11T22:54:11.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-140 /etc/passwd 2020-03-11T22:54:11.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.368 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-141 /etc/passwd 2020-03-11T22:54:11.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.418 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-142 /etc/passwd 2020-03-11T22:54:11.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.508 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-143 /etc/passwd 2020-03-11T22:54:11.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-144 /etc/passwd 2020-03-11T22:54:11.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.626 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-145 /etc/passwd 2020-03-11T22:54:11.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.716 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-146 /etc/passwd 2020-03-11T22:54:11.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.809 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-147 /etc/passwd 2020-03-11T22:54:11.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.860 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-148 /etc/passwd 2020-03-11T22:54:11.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:11.935 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-149 /etc/passwd 2020-03-11T22:54:12.025 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:61: _common_test: for j in $(seq 1 $objects) 2020-03-11T22:54:12.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:63: _common_test: rados -p test put obj-150 /etc/passwd 2020-03-11T22:54:12.077 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:54:12.078 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: jq '.pg_stats[0].up[]' 2020-03-11T22:54:12.403 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:54:12.411 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:67: _common_test: ceph osd out 1 0 2 2020-03-11T22:54:12.894 INFO:tasks.workunit.client.0.smithi155.stderr:osd.1 is already out. osd.0 is already out. osd.2 is already out. 2020-03-11T22:54:12.902 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:68: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:54:12.903 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:74: _common_test: sleep 1 2020-03-11T22:54:13.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:75: _common_test: wait_for_clean 2020-03-11T22:54:13.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1579: wait_for_clean: local cmd= 2020-03-11T22:54:13.904 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1580: wait_for_clean: local num_active_clean=-1 2020-03-11T22:54:13.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1581: wait_for_clean: local cur_active_clean 2020-03-11T22:54:13.905 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: delays=($(get_timeout_delays $WAIT_FOR_CLEAN_TIMEOUT .1)) 2020-03-11T22:54:13.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: get_timeout_delays 90 .1 2020-03-11T22:54:13.906 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: shopt -q -o xtrace 2020-03-11T22:54:13.906 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: echo true 2020-03-11T22:54:13.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1525: get_timeout_delays: local trace=true 2020-03-11T22:54:13.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: true 2020-03-11T22:54:13.906 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: get_timeout_delays: shopt -u -o xtrace 2020-03-11T22:54:13.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1582: wait_for_clean: local -a delays 2020-03-11T22:54:13.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1583: wait_for_clean: local -i loop=0 2020-03-11T22:54:13.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1585: wait_for_clean: flush_pg_stats 2020-03-11T22:54:13.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2066: flush_pg_stats: local timeout=300 2020-03-11T22:54:13.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ceph osd ls 2020-03-11T22:54:14.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: flush_pg_stats: ids='0 2020-03-11T22:54:14.347 INFO:tasks.workunit.client.0.smithi155.stderr:1 2020-03-11T22:54:14.347 INFO:tasks.workunit.client.0.smithi155.stderr:2 2020-03-11T22:54:14.347 INFO:tasks.workunit.client.0.smithi155.stderr:3 2020-03-11T22:54:14.347 INFO:tasks.workunit.client.0.smithi155.stderr:4 2020-03-11T22:54:14.347 INFO:tasks.workunit.client.0.smithi155.stderr:5' 2020-03-11T22:54:14.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: flush_pg_stats: seqs= 2020-03-11T22:54:14.348 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:14.348 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.0 flush_pg_stats 2020-03-11T22:54:14.469 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=21474836493 2020-03-11T22:54:14.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493' 2020-03-11T22:54:14.470 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:14.470 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.1 flush_pg_stats 2020-03-11T22:54:14.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=38654705676 2020-03-11T22:54:14.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676' 2020-03-11T22:54:14.595 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:14.595 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.2 flush_pg_stats 2020-03-11T22:54:14.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=55834574858 2020-03-11T22:54:14.719 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-55834574858' 2020-03-11T22:54:14.720 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:14.720 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.3 flush_pg_stats 2020-03-11T22:54:14.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=73014444041 2020-03-11T22:54:14.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-55834574858 3-73014444041' 2020-03-11T22:54:14.844 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:14.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.4 flush_pg_stats 2020-03-11T22:54:14.967 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=94489280520 2020-03-11T22:54:14.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-55834574858 3-73014444041 4-94489280520' 2020-03-11T22:54:14.968 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2070: flush_pg_stats: for osd in $ids 2020-03-11T22:54:14.968 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: ceph tell osd.5 flush_pg_stats 2020-03-11T22:54:15.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: flush_pg_stats: seq=115964116999 2020-03-11T22:54:15.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2072: flush_pg_stats: seqs=' 0-21474836493 1-38654705676 2-55834574858 3-73014444041 4-94489280520 5-115964116999' 2020-03-11T22:54:15.092 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:15.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 0-21474836493 2020-03-11T22:54:15.093 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:15.093 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=0 2020-03-11T22:54:15.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 0-21474836493 2020-03-11T22:54:15.094 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:15.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=21474836493 2020-03-11T22:54:15.095 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.0 seq 21474836493' 2020-03-11T22:54:15.096 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.0 seq 21474836493 2020-03-11T22:54:15.096 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:15.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836491 -lt 21474836493 2020-03-11T22:54:15.449 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2080: flush_pg_stats: sleep 1 2020-03-11T22:54:16.450 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2081: flush_pg_stats: '[' 300 -eq 0 ']' 2020-03-11T22:54:16.451 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 0 2020-03-11T22:54:16.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 21474836493 -lt 21474836493 2020-03-11T22:54:16.808 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:16.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 1-38654705676 2020-03-11T22:54:16.809 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:16.810 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=1 2020-03-11T22:54:16.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 1-38654705676 2020-03-11T22:54:16.811 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:16.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=38654705676 2020-03-11T22:54:16.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.1 seq 38654705676' 2020-03-11T22:54:16.812 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.1 seq 38654705676 2020-03-11T22:54:16.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 1 2020-03-11T22:54:17.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 38654705677 -lt 38654705676 2020-03-11T22:54:17.271 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:17.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 2-55834574858 2020-03-11T22:54:17.272 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:17.273 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=2 2020-03-11T22:54:17.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 2-55834574858 2020-03-11T22:54:17.274 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:17.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=55834574858 2020-03-11T22:54:17.275 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.2 seq 55834574858' 2020-03-11T22:54:17.275 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.2 seq 55834574858 2020-03-11T22:54:17.275 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 2 2020-03-11T22:54:17.621 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 55834574858 -lt 55834574858 2020-03-11T22:54:17.622 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:17.622 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 3-73014444041 2020-03-11T22:54:17.623 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:17.623 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=3 2020-03-11T22:54:17.624 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 3-73014444041 2020-03-11T22:54:17.624 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:17.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=73014444041 2020-03-11T22:54:17.625 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.3 seq 73014444041' 2020-03-11T22:54:17.625 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.3 seq 73014444041 2020-03-11T22:54:17.626 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 3 2020-03-11T22:54:17.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 73014444041 -lt 73014444041 2020-03-11T22:54:17.973 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:17.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 4-94489280520 2020-03-11T22:54:17.974 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:17.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=4 2020-03-11T22:54:17.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 4-94489280520 2020-03-11T22:54:17.976 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:17.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=94489280520 2020-03-11T22:54:17.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.4 seq 94489280520' 2020-03-11T22:54:17.977 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.4 seq 94489280520 2020-03-11T22:54:17.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 4 2020-03-11T22:54:18.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 94489280521 -lt 94489280520 2020-03-11T22:54:18.331 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2075: flush_pg_stats: for s in $seqs 2020-03-11T22:54:18.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: echo 5-115964116999 2020-03-11T22:54:18.332 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: cut -d - -f 1 2020-03-11T22:54:18.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2076: flush_pg_stats: osd=5 2020-03-11T22:54:18.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: echo 5-115964116999 2020-03-11T22:54:18.334 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: cut -d - -f 2 2020-03-11T22:54:18.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2077: flush_pg_stats: seq=115964116999 2020-03-11T22:54:18.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2078: flush_pg_stats: echo 'waiting osd.5 seq 115964116999' 2020-03-11T22:54:18.335 INFO:tasks.workunit.client.0.smithi155.stdout:waiting osd.5 seq 115964116999 2020-03-11T22:54:18.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: ceph osd last-stat-seq 5 2020-03-11T22:54:18.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2079: flush_pg_stats: test 115964116999 -lt 115964116999 2020-03-11T22:54:18.683 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: get_num_pgs 2020-03-11T22:54:18.683 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:18.684 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:19.152 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1586: wait_for_clean: test 1 == 0 2020-03-11T22:54:19.153 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:19.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:19.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:19.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:19.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:19.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:19.154 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:19.555 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:19.555 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:19.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:19.556 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:20.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:20.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' -1 2020-03-11T22:54:20.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1597: wait_for_clean: loop=0 2020-03-11T22:54:20.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1598: wait_for_clean: num_active_clean=0 2020-03-11T22:54:20.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:20.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:20.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:20.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:20.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:20.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:20.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:20.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:20.103 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:20.104 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:20.440 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:20.440 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:20.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:20.441 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:20.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:20.888 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:20.889 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:20.889 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:20.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:21.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=18005 2020-03-11T22:54:21.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 18005 '!=' null 2020-03-11T22:54:21.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:21.406 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:21.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:21.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:21.507 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:21.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:21.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:21.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:21.508 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:21.509 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:21.509 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:21.840 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:21.841 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:21.841 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:21.842 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:22.289 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:22.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:22.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:22.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:22.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:22.290 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:22.291 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:22.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:22.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:22.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14584 2020-03-11T22:54:22.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14584 '!=' null 2020-03-11T22:54:22.742 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:22.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:22.743 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:22.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:22.843 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:22.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:22.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:22.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:22.844 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:22.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:22.845 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:23.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:23.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:23.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:23.179 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:23.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:23.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:23.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:23.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:23.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:23.630 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:23.631 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:23.631 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:23.631 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:24.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=14584 2020-03-11T22:54:24.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 14584 '!=' null 2020-03-11T22:54:24.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:24.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:24.102 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:24.202 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:24.203 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:24.203 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:24.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:24.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:24.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:24.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:24.204 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:24.534 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:24.535 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:24.535 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:24.535 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:24.975 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:24.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:24.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:24.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:24.976 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:24.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:24.977 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:24.977 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:24.978 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:25.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=31161 2020-03-11T22:54:25.428 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 31161 '!=' null 2020-03-11T22:54:25.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:25.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:25.429 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:25.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:25.529 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:25.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:25.530 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:25.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:25.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:25.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:25.531 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:25.864 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:25.865 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:25.865 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:25.865 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:26.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:26.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:26.333 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:26.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:26.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:26.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:26.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:26.335 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:26.335 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:26.778 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27597 2020-03-11T22:54:26.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27597 '!=' null 2020-03-11T22:54:26.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:26.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:26.779 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:26.878 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:26.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:26.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:26.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:26.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:26.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:26.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:26.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:27.213 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:27.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:27.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:27.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:27.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:27.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:27.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:27.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:27.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:27.663 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:27.664 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:27.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:27.664 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:28.111 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27597 2020-03-11T22:54:28.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27597 '!=' null 2020-03-11T22:54:28.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:28.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:28.112 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:28.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:28.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:28.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:28.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:28.213 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:28.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:28.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:28.214 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:28.549 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:28.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:28.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:28.551 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:28.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:28.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:28.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:28.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:28.997 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:28.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:28.998 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:28.998 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:28.999 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:29.445 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=27598 2020-03-11T22:54:29.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 27598 '!=' null 2020-03-11T22:54:29.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:29.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:29.446 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:29.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:29.546 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:29.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:29.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:29.547 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:29.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:29.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:29.548 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:29.879 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:29.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:29.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:29.880 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:30.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:30.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:30.325 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:30.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:30.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:30.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:30.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:30.327 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:30.327 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:30.771 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=44848 2020-03-11T22:54:30.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 44848 '!=' null 2020-03-11T22:54:30.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:30.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:30.772 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:30.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:30.872 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:30.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:30.873 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:30.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:30.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:30.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:30.874 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:31.210 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:31.211 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:31.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:31.212 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:31.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:31.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:31.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:31.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:31.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:31.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:31.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:31.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:31.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:32.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=44848 2020-03-11T22:54:32.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 44848 '!=' null 2020-03-11T22:54:32.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:32.113 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:32.114 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:32.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:32.214 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:32.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:32.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:32.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:32.215 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:32.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:32.216 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:32.548 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:32.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:32.549 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:32.550 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:33.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:33.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:33.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:33.001 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:33.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:33.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:33.002 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:33.002 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:33.003 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:33.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33923 2020-03-11T22:54:33.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33923 '!=' null 2020-03-11T22:54:33.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:33.458 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:33.459 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:33.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:33.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:33.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:33.559 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:33.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:33.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:33.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:33.560 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:33.891 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:33.891 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:33.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:33.892 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:34.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:34.340 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:34.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:34.341 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:34.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=50023 2020-03-11T22:54:34.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 50023 '!=' null 2020-03-11T22:54:34.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:34.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:34.822 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:34.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:34.922 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:34.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:34.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:34.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:34.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:34.923 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:34.924 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:35.256 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:35.257 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:35.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:35.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:35.708 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:35.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:35.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:35.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:35.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:35.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:35.709 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:35.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:35.710 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:36.156 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=50023 2020-03-11T22:54:36.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 50023 '!=' null 2020-03-11T22:54:36.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:36.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:36.157 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:36.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:36.257 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:36.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:36.258 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:36.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:36.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:36.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:36.259 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:36.591 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:36.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:36.592 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:36.593 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:37.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:37.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:37.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:37.036 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:37.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:37.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:37.037 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:37.037 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:37.038 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:37.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33348 2020-03-11T22:54:37.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33348 '!=' null 2020-03-11T22:54:37.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:37.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:37.484 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:37.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:37.585 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:37.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:37.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:37.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:37.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:37.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:37.586 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:37.920 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:37.921 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:37.921 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:37.921 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:38.370 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:38.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:38.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:38.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:38.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:38.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:38.371 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:38.372 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:38.372 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:38.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33348 2020-03-11T22:54:38.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33348 '!=' null 2020-03-11T22:54:38.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:38.845 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:38.846 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:38.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:38.946 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:38.947 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:38.947 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:38.947 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:38.947 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:38.947 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:38.947 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:39.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:39.311 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:39.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:39.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:39.773 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:39.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:39.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:39.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:39.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:39.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:39.774 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:39.775 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:39.775 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:40.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=33348 2020-03-11T22:54:40.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 33348 '!=' null 2020-03-11T22:54:40.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:40.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:40.221 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:40.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:40.322 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:40.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:40.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:40.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:40.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:40.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:40.323 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:40.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:40.660 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:40.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:40.661 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:41.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:41.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:41.106 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:41.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:41.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:41.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:41.107 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:41.107 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:41.108 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:41.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=49448 2020-03-11T22:54:41.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 49448 '!=' null 2020-03-11T22:54:41.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:41.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:41.552 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:41.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:41.652 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:41.653 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:41.653 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:41.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:41.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:41.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:41.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:41.988 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:41.989 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:41.989 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:41.990 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:42.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:42.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:42.437 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:42.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:42.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:42.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:42.438 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:42.439 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:42.439 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:42.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24149 2020-03-11T22:54:42.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24149 '!=' null 2020-03-11T22:54:42.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:42.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:42.934 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:43.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:43.034 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:43.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:43.035 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:43.036 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:43.036 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:43.036 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:43.036 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:43.365 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:43.366 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:43.366 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:43.367 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:43.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:43.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:43.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:43.811 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:43.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:43.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:43.812 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:43.812 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:43.813 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:44.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24149 2020-03-11T22:54:44.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24149 '!=' null 2020-03-11T22:54:44.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:44.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:44.255 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:44.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:44.355 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:44.356 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:44.356 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:44.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:44.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:44.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:44.357 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:44.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:44.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:44.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:44.686 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:45.125 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:45.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:45.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:45.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:45.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:45.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:45.126 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:45.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:45.127 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:45.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=24149 2020-03-11T22:54:45.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 24149 '!=' null 2020-03-11T22:54:45.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:45.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:45.594 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:45.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:45.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:45.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:45.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:45.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:45.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:45.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:45.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:46.026 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=0 2020-03-11T22:54:46.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:46.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:46.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:46.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 0 = 1 2020-03-11T22:54:46.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1596: wait_for_clean: test 0 '!=' 0 2020-03-11T22:54:46.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1599: wait_for_clean: get_is_making_recovery_progress 2020-03-11T22:54:46.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1271: get_is_making_recovery_progress: local recovery_progress 2020-03-11T22:54:46.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1272: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2020-03-11T22:54:46.515 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1273: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2020-03-11T22:54:46.516 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1274: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2020-03-11T22:54:46.516 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: ceph --format json status 2020-03-11T22:54:46.516 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2020-03-11T22:54:46.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1276: get_is_making_recovery_progress: local progress=37948 2020-03-11T22:54:46.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1277: get_is_making_recovery_progress: test 37948 '!=' null 2020-03-11T22:54:46.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1600: wait_for_clean: loop=0 2020-03-11T22:54:46.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1606: wait_for_clean: eval 2020-03-11T22:54:46.957 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1607: wait_for_clean: sleep 0.1 2020-03-11T22:54:47.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1608: wait_for_clean: loop+=1 2020-03-11T22:54:47.057 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1590: wait_for_clean: true 2020-03-11T22:54:47.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: get_num_active_clean 2020-03-11T22:54:47.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: get_num_active_clean: local expression 2020-03-11T22:54:47.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2020-03-11T22:54:47.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: get_num_active_clean: expression+='select(contains("stale") | not)' 2020-03-11T22:54:47.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: get_num_active_clean: ceph --format json pg dump pgs 2020-03-11T22:54:47.059 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2020-03-11T22:54:47.388 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: wait_for_clean: cur_active_clean=1 2020-03-11T22:54:47.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: get_num_pgs 2020-03-11T22:54:47.389 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: ceph --format json status 2020-03-11T22:54:47.390 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1362: get_num_pgs: jq .pgmap.num_pgs 2020-03-11T22:54:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: test 1 = 1 2020-03-11T22:54:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1595: wait_for_clean: break 2020-03-11T22:54:47.831 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1610: wait_for_clean: return 0 2020-03-11T22:54:47.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: ceph pg dump pgs --format=json 2020-03-11T22:54:47.832 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: jq '.pg_stats[0].up_primary' 2020-03-11T22:54:48.162 INFO:tasks.workunit.client.0.smithi155.stderr:dumped pgs 2020-03-11T22:54:48.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:77: _common_test: newprimary=4 2020-03-11T22:54:48.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:78: _common_test: kill_daemons 2020-03-11T22:54:48.171 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:54:48.171 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:54:48.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:54:48.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:54:48.172 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:54:48.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:54:48.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:80: _common_test: ERRORS=0 2020-03-11T22:54:48.482 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: _objectstore_tool_nodown td/osd-backfill-recovery-log 4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:54:48.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:81: _common_test: tee td/osd-backfill-recovery-log/result.log 2020-03-11T22:54:48.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1194: _objectstore_tool_nodown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:48.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: _objectstore_tool_nodown: shift 2020-03-11T22:54:48.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1196: _objectstore_tool_nodown: local id=4 2020-03-11T22:54:48.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: _objectstore_tool_nodown: shift 2020-03-11T22:54:48.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: _objectstore_tool_nodown: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:54:48.483 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/osd-backfill-recovery-log/4 --no-mon-config --pgid 1.0 --op log 2020-03-11T22:54:49.616 INFO:tasks.workunit.client.0.smithi155.stdout:{ 2020-03-11T22:54:49.617 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_log_t": { 2020-03-11T22:54:49.617 INFO:tasks.workunit.client.0.smithi155.stdout: "head": "31'150", 2020-03-11T22:54:49.617 INFO:tasks.workunit.client.0.smithi155.stdout: "tail": "31'148", 2020-03-11T22:54:49.617 INFO:tasks.workunit.client.0.smithi155.stdout: "log": [ 2020-03-11T22:54:49.617 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:54:49.617 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "modify", 2020-03-11T22:54:49.618 INFO:tasks.workunit.client.0.smithi155.stdout: "object": "1:e9c179f9:::obj-149:head", 2020-03-11T22:54:49.618 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "31'149", 2020-03-11T22:54:49.618 INFO:tasks.workunit.client.0.smithi155.stdout: "prior_version": "0'0", 2020-03-11T22:54:49.618 INFO:tasks.workunit.client.0.smithi155.stdout: "reqid": "client.4635.0:1", 2020-03-11T22:54:49.618 INFO:tasks.workunit.client.0.smithi155.stdout: "extra_reqids": [], 2020-03-11T22:54:49.618 INFO:tasks.workunit.client.0.smithi155.stdout: "mtime": "2020-03-11T22:54:11.954108+0000", 2020-03-11T22:54:49.618 INFO:tasks.workunit.client.0.smithi155.stdout: "return_code": 0, 2020-03-11T22:54:49.619 INFO:tasks.workunit.client.0.smithi155.stdout: "mod_desc": { 2020-03-11T22:54:49.619 INFO:tasks.workunit.client.0.smithi155.stdout: "object_mod_desc": { 2020-03-11T22:54:49.619 INFO:tasks.workunit.client.0.smithi155.stdout: "can_local_rollback": false, 2020-03-11T22:54:49.619 INFO:tasks.workunit.client.0.smithi155.stdout: "rollback_info_completed": false, 2020-03-11T22:54:49.619 INFO:tasks.workunit.client.0.smithi155.stdout: "ops": [] 2020-03-11T22:54:49.619 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:54:49.619 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:54:49.620 INFO:tasks.workunit.client.0.smithi155.stdout: "clean_regions": { 2020-03-11T22:54:49.620 INFO:tasks.workunit.client.0.smithi155.stdout: "object_clean_regions": { 2020-03-11T22:54:49.620 INFO:tasks.workunit.client.0.smithi155.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:54:49.620 INFO:tasks.workunit.client.0.smithi155.stdout: "clean_omap": true, 2020-03-11T22:54:49.620 INFO:tasks.workunit.client.0.smithi155.stdout: "new_object": false 2020-03-11T22:54:49.620 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:54:49.620 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:54:49.621 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:54:49.621 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:54:49.621 INFO:tasks.workunit.client.0.smithi155.stdout: "op": "modify", 2020-03-11T22:54:49.621 INFO:tasks.workunit.client.0.smithi155.stdout: "object": "1:cb39fd60:::obj-150:head", 2020-03-11T22:54:49.621 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "31'150", 2020-03-11T22:54:49.621 INFO:tasks.workunit.client.0.smithi155.stdout: "prior_version": "0'0", 2020-03-11T22:54:49.621 INFO:tasks.workunit.client.0.smithi155.stdout: "reqid": "client.4638.0:1", 2020-03-11T22:54:49.621 INFO:tasks.workunit.client.0.smithi155.stdout: "extra_reqids": [], 2020-03-11T22:54:49.622 INFO:tasks.workunit.client.0.smithi155.stdout: "mtime": "2020-03-11T22:54:12.045104+0000", 2020-03-11T22:54:49.622 INFO:tasks.workunit.client.0.smithi155.stdout: "return_code": 0, 2020-03-11T22:54:49.622 INFO:tasks.workunit.client.0.smithi155.stdout: "mod_desc": { 2020-03-11T22:54:49.622 INFO:tasks.workunit.client.0.smithi155.stdout: "object_mod_desc": { 2020-03-11T22:54:49.622 INFO:tasks.workunit.client.0.smithi155.stdout: "can_local_rollback": false, 2020-03-11T22:54:49.622 INFO:tasks.workunit.client.0.smithi155.stdout: "rollback_info_completed": false, 2020-03-11T22:54:49.622 INFO:tasks.workunit.client.0.smithi155.stdout: "ops": [] 2020-03-11T22:54:49.623 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:54:49.623 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:54:49.623 INFO:tasks.workunit.client.0.smithi155.stdout: "clean_regions": { 2020-03-11T22:54:49.623 INFO:tasks.workunit.client.0.smithi155.stdout: "object_clean_regions": { 2020-03-11T22:54:49.623 INFO:tasks.workunit.client.0.smithi155.stdout: "clean_offsets": "[6900~18446744073709544715]", 2020-03-11T22:54:49.623 INFO:tasks.workunit.client.0.smithi155.stdout: "clean_omap": true, 2020-03-11T22:54:49.623 INFO:tasks.workunit.client.0.smithi155.stdout: "new_object": false 2020-03-11T22:54:49.624 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:54:49.624 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:54:49.624 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:54:49.624 INFO:tasks.workunit.client.0.smithi155.stdout: ], 2020-03-11T22:54:49.624 INFO:tasks.workunit.client.0.smithi155.stdout: "dups": [ 2020-03-11T22:54:49.624 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:54:49.624 INFO:tasks.workunit.client.0.smithi155.stdout: "reqid": "client.4611.0:1", 2020-03-11T22:54:49.624 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "31'141", 2020-03-11T22:54:49.625 INFO:tasks.workunit.client.0.smithi155.stdout: "user_version": "141", 2020-03-11T22:54:49.625 INFO:tasks.workunit.client.0.smithi155.stdout: "return_code": "0" 2020-03-11T22:54:49.625 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:54:49.625 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:54:49.625 INFO:tasks.workunit.client.0.smithi155.stdout: "reqid": "client.4614.0:1", 2020-03-11T22:54:49.625 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "31'142", 2020-03-11T22:54:49.625 INFO:tasks.workunit.client.0.smithi155.stdout: "user_version": "142", 2020-03-11T22:54:49.625 INFO:tasks.workunit.client.0.smithi155.stdout: "return_code": "0" 2020-03-11T22:54:49.626 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:54:49.626 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:54:49.626 INFO:tasks.workunit.client.0.smithi155.stdout: "reqid": "client.4617.0:1", 2020-03-11T22:54:49.626 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "31'143", 2020-03-11T22:54:49.626 INFO:tasks.workunit.client.0.smithi155.stdout: "user_version": "143", 2020-03-11T22:54:49.626 INFO:tasks.workunit.client.0.smithi155.stdout: "return_code": "0" 2020-03-11T22:54:49.626 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:54:49.627 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:54:49.627 INFO:tasks.workunit.client.0.smithi155.stdout: "reqid": "client.4620.0:1", 2020-03-11T22:54:49.627 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "31'144", 2020-03-11T22:54:49.627 INFO:tasks.workunit.client.0.smithi155.stdout: "user_version": "144", 2020-03-11T22:54:49.627 INFO:tasks.workunit.client.0.smithi155.stdout: "return_code": "0" 2020-03-11T22:54:49.627 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:54:49.627 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:54:49.627 INFO:tasks.workunit.client.0.smithi155.stdout: "reqid": "client.4623.0:1", 2020-03-11T22:54:49.628 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "31'145", 2020-03-11T22:54:49.628 INFO:tasks.workunit.client.0.smithi155.stdout: "user_version": "145", 2020-03-11T22:54:49.628 INFO:tasks.workunit.client.0.smithi155.stdout: "return_code": "0" 2020-03-11T22:54:49.628 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:54:49.628 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:54:49.628 INFO:tasks.workunit.client.0.smithi155.stdout: "reqid": "client.4626.0:1", 2020-03-11T22:54:49.628 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "31'146", 2020-03-11T22:54:49.629 INFO:tasks.workunit.client.0.smithi155.stdout: "user_version": "146", 2020-03-11T22:54:49.629 INFO:tasks.workunit.client.0.smithi155.stdout: "return_code": "0" 2020-03-11T22:54:49.629 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:54:49.629 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:54:49.629 INFO:tasks.workunit.client.0.smithi155.stdout: "reqid": "client.4629.0:1", 2020-03-11T22:54:49.629 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "31'147", 2020-03-11T22:54:49.629 INFO:tasks.workunit.client.0.smithi155.stdout: "user_version": "147", 2020-03-11T22:54:49.629 INFO:tasks.workunit.client.0.smithi155.stdout: "return_code": "0" 2020-03-11T22:54:49.630 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:54:49.630 INFO:tasks.workunit.client.0.smithi155.stdout: { 2020-03-11T22:54:49.630 INFO:tasks.workunit.client.0.smithi155.stdout: "reqid": "client.4632.0:1", 2020-03-11T22:54:49.630 INFO:tasks.workunit.client.0.smithi155.stdout: "version": "31'148", 2020-03-11T22:54:49.630 INFO:tasks.workunit.client.0.smithi155.stdout: "user_version": "148", 2020-03-11T22:54:49.630 INFO:tasks.workunit.client.0.smithi155.stdout: "return_code": "0" 2020-03-11T22:54:49.630 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:54:49.630 INFO:tasks.workunit.client.0.smithi155.stdout: ] 2020-03-11T22:54:49.631 INFO:tasks.workunit.client.0.smithi155.stdout: }, 2020-03-11T22:54:49.631 INFO:tasks.workunit.client.0.smithi155.stdout: "pg_missing_t": { 2020-03-11T22:54:49.631 INFO:tasks.workunit.client.0.smithi155.stdout: "missing": [], 2020-03-11T22:54:49.631 INFO:tasks.workunit.client.0.smithi155.stdout: "may_include_deletes": true 2020-03-11T22:54:49.631 INFO:tasks.workunit.client.0.smithi155.stdout: } 2020-03-11T22:54:49.631 INFO:tasks.workunit.client.0.smithi155.stdout:} 2020-03-11T22:54:49.632 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: jq '.pg_log_t.log | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:54:49.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:82: _common_test: LOGLEN=2 2020-03-11T22:54:49.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:83: _common_test: '[' 2 '!=' 2 ']' 2020-03-11T22:54:49.633 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: jq '.pg_log_t.dups | length' td/osd-backfill-recovery-log/result.log 2020-03-11T22:54:49.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:87: _common_test: DUPSLEN=8 2020-03-11T22:54:49.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:88: _common_test: '[' 8 '!=' 8 ']' 2020-03-11T22:54:49.633 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:92: _common_test: grep 'copy_up_to\|copy_after' td/osd-backfill-recovery-log/osd.0.log td/osd-backfill-recovery-log/osd.1.log td/osd-backfill-recovery-log/osd.2.log td/osd-backfill-recovery-log/osd.3.log td/osd-backfill-recovery-log/osd.4.log td/osd-backfill-recovery-log/osd.5.log 2020-03-11T22:54:49.634 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to max 2 2020-03-11T22:54:49.634 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to copy log version 31'150 2020-03-11T22:54:49.634 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to copy log version 31'149 2020-03-11T22:54:49.634 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:54:49.634 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'141 2020-03-11T22:54:49.634 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'142 2020-03-11T22:54:49.634 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'143 2020-03-11T22:54:49.635 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'144 2020-03-11T22:54:49.635 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'145 2020-03-11T22:54:49.635 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'146 2020-03-11T22:54:49.635 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'147 2020-03-11T22:54:49.635 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'148 2020-03-11T22:54:49.635 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to max 2 2020-03-11T22:54:49.635 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to copy log version 31'150 2020-03-11T22:54:49.636 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to copy log version 31'149 2020-03-11T22:54:49.636 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:54:49.636 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'141 2020-03-11T22:54:49.636 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'142 2020-03-11T22:54:49.636 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'143 2020-03-11T22:54:49.636 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'144 2020-03-11T22:54:49.636 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'145 2020-03-11T22:54:49.637 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'146 2020-03-11T22:54:49.637 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'147 2020-03-11T22:54:49.637 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'148 2020-03-11T22:54:49.637 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to max 2 2020-03-11T22:54:49.637 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to copy log version 31'150 2020-03-11T22:54:49.637 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to copy log version 31'149 2020-03-11T22:54:49.637 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:54:49.637 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'141 2020-03-11T22:54:49.638 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'142 2020-03-11T22:54:49.638 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'143 2020-03-11T22:54:49.645 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'144 2020-03-11T22:54:49.645 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'145 2020-03-11T22:54:49.645 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'146 2020-03-11T22:54:49.645 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'147 2020-03-11T22:54:49.645 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:13.558+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'148 2020-03-11T22:54:49.645 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:41.802+0000 7fbf65e22700 20 copy_after v 31'150 2020-03-11T22:54:49.646 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:41.802+0000 7fbf65e22700 20 copy_up_to/copy_after earliest_dup_version 141 2020-03-11T22:54:49.646 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:41.802+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'141 2020-03-11T22:54:49.646 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:41.802+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'142 2020-03-11T22:54:49.646 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:41.802+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'143 2020-03-11T22:54:49.646 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:41.802+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'144 2020-03-11T22:54:49.646 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:41.802+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'145 2020-03-11T22:54:49.646 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:41.802+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'146 2020-03-11T22:54:49.646 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:41.802+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'147 2020-03-11T22:54:49.647 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:41.802+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'148 2020-03-11T22:54:49.647 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:41.802+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'149 2020-03-11T22:54:49.647 INFO:tasks.workunit.client.0.smithi155.stdout:td/osd-backfill-recovery-log/osd.0.log:2020-03-11T22:54:41.802+0000 7fbf65e22700 20 copy_up_to/copy_after copy dup from log version 31'150 2020-03-11T22:54:49.647 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:93: _common_test: rm -f td/osd-backfill-recovery-log/result.log 2020-03-11T22:54:49.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:94: _common_test: '[' 0 '!=' 0 ']' 2020-03-11T22:54:49.648 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:34: run: teardown td/osd-backfill-recovery-log 2020-03-11T22:54:49.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:49.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:54:49.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:54:49.649 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:54:49.649 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:54:49.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:54:49.649 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:54:49.650 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:54:49.654 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:54:49.654 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:54:49.655 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:54:49.655 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:54:49.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:54:49.656 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:54:49.656 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:54:49.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:54:49.658 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:54:49.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:54:49.659 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:54:49.659 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:54:49.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:54:49.660 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:54:49.678 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:54:49.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:49.679 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:54:49.679 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76480 2020-03-11T22:54:49.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:54:49.680 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:54:49.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:31: run: for func in $funcs 2020-03-11T22:54:49.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:32: run: setup td/osd-backfill-recovery-log 2020-03-11T22:54:49.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:131: setup: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:49.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:132: setup: teardown td/osd-backfill-recovery-log 2020-03-11T22:54:49.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:164: teardown: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:49.681 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: local dumplogs= 2020-03-11T22:54:49.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:166: teardown: kill_daemons td/osd-backfill-recovery-log KILL 2020-03-11T22:54:49.682 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: shopt -q -o xtrace 2020-03-11T22:54:49.682 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: echo true 2020-03-11T22:54:49.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:345: kill_daemons: local trace=true 2020-03-11T22:54:49.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: true 2020-03-11T22:54:49.682 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:346: kill_daemons: shopt -u -o xtrace 2020-03-11T22:54:49.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:362: kill_daemons: return 0 2020-03-11T22:54:49.683 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: uname 2020-03-11T22:54:49.683 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:167: teardown: '[' Linux '!=' FreeBSD ']' 2020-03-11T22:54:49.683 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: stat -f -c %T . 2020-03-11T22:54:49.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:168: teardown: '[' ext2/ext3 == btrfs ']' 2020-03-11T22:54:49.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: local cores=no 2020-03-11T22:54:49.685 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: sysctl -n kernel.core_pattern 2020-03-11T22:54:49.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:172: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:54:49.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:174: teardown: '[' / = '|' ']' 2020-03-11T22:54:49.686 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: grep -q '^core\|core$' 2020-03-11T22:54:49.687 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2020-03-11T22:54:49.687 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: ls /home/ubuntu/cephtest/archive/coredump 2020-03-11T22:54:49.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: '[' no = yes -o '' = 1 ']' 2020-03-11T22:54:49.688 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: rm -fr td/osd-backfill-recovery-log 2020-03-11T22:54:49.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: get_asok_dir 2020-03-11T22:54:49.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:49.689 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:54:49.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:199: teardown: rm -rf /tmp/ceph-asok.76480 2020-03-11T22:54:49.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:200: teardown: '[' no = yes ']' 2020-03-11T22:54:49.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:207: teardown: return 0 2020-03-11T22:54:49.690 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:133: setup: mkdir -p td/osd-backfill-recovery-log 2020-03-11T22:54:49.691 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: get_asok_dir 2020-03-11T22:54:49.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:49.692 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:54:49.692 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:134: setup: mkdir -p /tmp/ceph-asok.76480 2020-03-11T22:54:49.694 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: ulimit -n 2020-03-11T22:54:49.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:135: setup: '[' 4096 -le 1024 ']' 2020-03-11T22:54:49.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:138: setup: '[' -z '' ']' 2020-03-11T22:54:49.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:139: setup: trap 'teardown td/osd-backfill-recovery-log 1' TERM HUP INT 2020-03-11T22:54:49.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:33: run: TEST_backfill_log_2 td/osd-backfill-recovery-log 2020-03-11T22:54:49.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:111: TEST_backfill_log_2: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:49.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:113: TEST_backfill_log_2: _common_test td/osd-backfill-recovery-log '--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2 148 150 2020-03-11T22:54:49.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:40: _common_test: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:49.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:41: _common_test: local 'extra_opts=--osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:49.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:42: _common_test: local loglen=2 2020-03-11T22:54:49.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:43: _common_test: local dupslen=148 2020-03-11T22:54:49.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:44: _common_test: local objects=150 2020-03-11T22:54:49.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:45: _common_test: local moreobjects=0 2020-03-11T22:54:49.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:47: _common_test: local OSDS=6 2020-03-11T22:54:49.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:49: _common_test: run_mon td/osd-backfill-recovery-log a 2020-03-11T22:54:49.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:448: run_mon: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:49.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:449: run_mon: shift 2020-03-11T22:54:49.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:450: run_mon: local id=a 2020-03-11T22:54:49.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:451: run_mon: shift 2020-03-11T22:54:49.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:452: run_mon: local data=td/osd-backfill-recovery-log/a 2020-03-11T22:54:49.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:455: run_mon: ceph-mon --id a --mkfs --mon-data=td/osd-backfill-recovery-log/a --run-dir=td/osd-backfill-recovery-log 2020-03-11T22:54:50.025 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: get_asok_path 2020-03-11T22:54:50.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:50.026 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:50.026 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:50.026 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:50.027 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:54:50.027 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:54:50.027 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:462: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/osd-backfill-recovery-log/a '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --mon-cluster-log-file=td/osd-backfill-recovery-log/log --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mon-allow-pool-delete --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 2020-03-11T22:54:50.228 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: cat 2020-03-11T22:54:50.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a fsid 2020-03-11T22:54:50.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:54:50.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:54:50.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=fsid 2020-03-11T22:54:50.228 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .fsid 2020-03-11T22:54:50.229 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:54:50.229 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:54:50.229 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:54:50.229 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:50.229 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:50.229 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:54:50.229 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76480/ceph-mon.a.asok 2020-03-11T22:54:50.230 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:54:50.230 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76480/ceph-mon.a.asok config get fsid 2020-03-11T22:54:50.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:485: run_mon: get_config mon a mon_host 2020-03-11T22:54:50.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: get_config: local daemon=mon 2020-03-11T22:54:50.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: get_config: local id=a 2020-03-11T22:54:50.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: get_config: local config=mon_host 2020-03-11T22:54:50.320 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: get_config: jq -r .mon_host 2020-03-11T22:54:50.320 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: get_asok_path mon.a 2020-03-11T22:54:50.320 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name=mon.a 2020-03-11T22:54:50.320 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n mon.a ']' 2020-03-11T22:54:50.320 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: get_asok_dir 2020-03-11T22:54:50.321 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:50.321 INFO:tasks.workunit.client.0.smithi155.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:54:50.321 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:118: get_asok_path: echo /tmp/ceph-asok.76480/ceph-mon.a.asok 2020-03-11T22:54:50.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: CEPH_ARGS= 2020-03-11T22:54:50.321 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: get_config: ceph --format json daemon /tmp/ceph-asok.76480/ceph-mon.a.asok config get mon_host 2020-03-11T22:54:50.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:50: _common_test: run_mgr td/osd-backfill-recovery-log x 2020-03-11T22:54:50.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:551: run_mgr: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:50.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: shift 2020-03-11T22:54:50.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:553: run_mgr: local id=x 2020-03-11T22:54:50.407 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:554: run_mgr: shift 2020-03-11T22:54:50.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:555: run_mgr: local data=td/osd-backfill-recovery-log/x 2020-03-11T22:54:50.408 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:557: run_mgr: ceph config set mgr mgr/devicehealth/enable_monitoring off --force 2020-03-11T22:54:51.289 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: get_asok_path 2020-03-11T22:54:51.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:51.291 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:51.292 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:51.292 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:51.292 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:54:51.292 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:54:51.293 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:54:51.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:559: run_mgr: ceph-mgr --id x --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --osd_pg_log_dups_tracked=10 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/osd-backfill-recovery-log/x '--log-file=td/osd-backfill-recovery-log/$name.log' '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --run-dir=td/osd-backfill-recovery-log '--pid-file=td/osd-backfill-recovery-log/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2020-03-11T22:54:51.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:51: _common_test: export CEPH_ARGS 2020-03-11T22:54:51.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: export 'EXTRA_OPTS= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:51.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:52: _common_test: EXTRA_OPTS=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:51.308 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: expr 6 - 1 2020-03-11T22:54:51.309 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: seq 0 5 2020-03-11T22:54:51.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:51.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 0 2020-03-11T22:54:51.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:51.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:51.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=0 2020-03-11T22:54:51.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:51.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/0 2020-03-11T22:54:51.314 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:51.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:51.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:51.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:51.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/0' 2020-03-11T22:54:51.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/0/journal' 2020-03-11T22:54:51.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:51.315 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:51.316 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:51.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:51.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:51.316 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:51.316 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:51.316 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:51.317 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:54:51.317 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:54:51.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:54:51.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:51.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:51.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:51.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:51.317 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:51.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:51.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:51.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:51.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:51.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:51.318 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/0 2020-03-11T22:54:51.318 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:51.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=67b7b500-0cea-4a4d-9717-e2e698577d38 2020-03-11T22:54:51.319 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd0 67b7b500-0cea-4a4d-9717-e2e698577d38' 2020-03-11T22:54:51.319 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:51.319 INFO:tasks.workunit.client.0.smithi155.stdout:add osd0 67b7b500-0cea-4a4d-9717-e2e698577d38 2020-03-11T22:54:51.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQA7bGlebBtRExAAqaIHr7UFwh391sQxxMasFg== 2020-03-11T22:54:51.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQA7bGlebBtRExAAqaIHr7UFwh391sQxxMasFg=="}' 2020-03-11T22:54:51.327 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 67b7b500-0cea-4a4d-9717-e2e698577d38 -i td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:54:51.675 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:54:51.684 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/0/new.json 2020-03-11T22:54:51.685 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 0 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQA7bGlebBtRExAAqaIHr7UFwh391sQxxMasFg== --osd-uuid 67b7b500-0cea-4a4d-9717-e2e698577d38 2020-03-11T22:54:51.710 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:51.706+0000 7f96f324aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:51.711 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:51.707+0000 7f96f324aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:51.713 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:51.709+0000 7f96f324aec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:51.713 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:51.710+0000 7f96f324aec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:54:51.714 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:51.710+0000 7f96f324aec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:54:51.714 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:51.710+0000 7f96f324aec0 -1 bluestore(td/osd-backfill-recovery-log/0/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/0/block: (2) No such file or directory 2020-03-11T22:54:51.767 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:51.763+0000 7f96f324aec0 -1 bluestore(td/osd-backfill-recovery-log/0) _read_fsid unparsable uuid 2020-03-11T22:54:54.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/0/keyring 2020-03-11T22:54:54.849 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:54:54.850 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd0 key to auth repository 2020-03-11T22:54:54.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd0 key to auth repository 2020-03-11T22:54:54.850 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:54:55.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.0 2020-03-11T22:54:55.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 0 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/0 --osd-journal=td/osd-backfill-recovery-log/0/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:55.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:54:55.292 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:54:55.292 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.0 2020-03-11T22:54:55.293 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:54:55.314 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:55.310+0000 7f5365225ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:55.315 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:55.311+0000 7f5365225ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:55.317 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:55.313+0000 7f5365225ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:55.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 0 2020-03-11T22:54:55.635 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:54:55.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=0 2020-03-11T22:54:55.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:54:55.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:54:55.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:55.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:54:55.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:54:55.636 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:55.637 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:54:55.834 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:55.830+0000 7f5365225ec0 -1 Falling back to public interface 2020-03-11T22:54:55.981 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:54:56.591 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:56.587+0000 7f5365225ec0 -1 osd.0 0 log_to_monitors {default=true} 2020-03-11T22:54:56.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:54:56.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:54:56.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:54:56.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.0 up' 2020-03-11T22:54:56.984 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:54:56.984 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:54:57.333 INFO:tasks.workunit.client.0.smithi155.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/85266,v1:127.0.0.1:6803/85266] [v2:127.0.0.1:6804/85266,v1:127.0.0.1:6805/85266] exists,up 67b7b500-0cea-4a4d-9717-e2e698577d38 2020-03-11T22:54:57.333 INFO:tasks.workunit.client.0.smithi155.stdout:add osd1 e1d55373-be95-4f69-b102-4800996084e7 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 1 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:54:57.334 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=1 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/1 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:54:57.335 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/1' 2020-03-11T22:54:57.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/1/journal' 2020-03-11T22:54:57.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:54:57.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:54:57.336 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:54:57.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:54:57.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:54:57.336 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:54:57.337 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:54:57.337 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:54:57.337 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:54:57.337 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:54:57.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:54:57.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:54:57.337 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:54:57.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:54:57.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:54:57.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:54:57.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:54:57.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:54:57.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:54:57.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:54:57.338 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:54:57.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/1 2020-03-11T22:54:57.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:54:57.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=e1d55373-be95-4f69-b102-4800996084e7 2020-03-11T22:54:57.339 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd1 e1d55373-be95-4f69-b102-4800996084e7' 2020-03-11T22:54:57.339 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:54:57.346 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBBbGle76eAFBAAVPuYDcLpMwTkFokvZNbpsg== 2020-03-11T22:54:57.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBBbGle76eAFBAAVPuYDcLpMwTkFokvZNbpsg=="}' 2020-03-11T22:54:57.347 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new e1d55373-be95-4f69-b102-4800996084e7 -i td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:54:57.884 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:54:57.894 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/1/new.json 2020-03-11T22:54:57.895 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 1 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBBbGle76eAFBAAVPuYDcLpMwTkFokvZNbpsg== --osd-uuid e1d55373-be95-4f69-b102-4800996084e7 2020-03-11T22:54:57.918 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:57.914+0000 7fadc8417ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:57.925 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:57.922+0000 7fadc8417ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:57.927 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:57.923+0000 7fadc8417ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:54:57.927 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:57.923+0000 7fadc8417ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:54:57.927 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:57.923+0000 7fadc8417ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:54:57.928 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:57.923+0000 7fadc8417ec0 -1 bluestore(td/osd-backfill-recovery-log/1/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/1/block: (2) No such file or directory 2020-03-11T22:54:58.050 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:54:58.047+0000 7fadc8417ec0 -1 bluestore(td/osd-backfill-recovery-log/1) _read_fsid unparsable uuid 2020-03-11T22:55:01.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/1/keyring 2020-03-11T22:55:01.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:01.180 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd1 key to auth repository 2020-03-11T22:55:01.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd1 key to auth repository 2020-03-11T22:55:01.181 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:01.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.1 2020-03-11T22:55:01.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 1 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/1 --osd-journal=td/osd-backfill-recovery-log/1/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:01.637 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:01.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:01.638 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:01.638 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.1 2020-03-11T22:55:01.658 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:01.655+0000 7f36608f3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:01.660 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:01.656+0000 7f36608f3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:01.662 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:01.659+0000 7f36608f3ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:01.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 1 2020-03-11T22:55:01.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:01.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=1 2020-03-11T22:55:01.982 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:01.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:01.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:01.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:01.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:01.983 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:55:01.983 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:55:02.178 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:02.175+0000 7f36608f3ec0 -1 Falling back to public interface 2020-03-11T22:55:02.324 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:02.891 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:02.887+0000 7f36608f3ec0 -1 osd.1 0 log_to_monitors {default=true} 2020-03-11T22:55:03.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:03.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:03.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:03.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:03.326 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.1 up' 2020-03-11T22:55:03.327 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:55:03.692 INFO:tasks.workunit.client.0.smithi155.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/85621,v1:127.0.0.1:6811/85621] [v2:127.0.0.1:6812/85621,v1:127.0.0.1:6813/85621] exists,up e1d55373-be95-4f69-b102-4800996084e7 2020-03-11T22:55:03.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:55:03.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:55:03.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:55:03.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:03.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 2 2020-03-11T22:55:03.693 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:03.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:03.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=2 2020-03-11T22:55:03.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:03.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/2 2020-03-11T22:55:03.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:03.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:03.694 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:03.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:03.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/2' 2020-03-11T22:55:03.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/2/journal' 2020-03-11T22:55:03.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:03.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:55:03.695 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:03.695 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:03.695 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:03.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:03.696 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:03.696 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:03.696 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:55:03.696 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:55:03.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:55:03.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:03.696 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:03.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:03.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:03.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:03.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:03.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:03.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:03.697 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:03.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:03.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/2 2020-03-11T22:55:03.698 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:03.698 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=9863531a-af37-463c-95a1-920808222a67 2020-03-11T22:55:03.699 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd2 9863531a-af37-463c-95a1-920808222a67' 2020-03-11T22:55:03.699 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:03.699 INFO:tasks.workunit.client.0.smithi155.stdout:add osd2 9863531a-af37-463c-95a1-920808222a67 2020-03-11T22:55:03.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBHbGlelNb4KRAAxAZclsJwwbbWoSEqyK4AZA== 2020-03-11T22:55:03.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBHbGlelNb4KRAAxAZclsJwwbbWoSEqyK4AZA=="}' 2020-03-11T22:55:03.707 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 9863531a-af37-463c-95a1-920808222a67 -i td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:55:04.169 INFO:tasks.workunit.client.0.smithi155.stdout:2 2020-03-11T22:55:04.179 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/2/new.json 2020-03-11T22:55:04.180 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 2 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBHbGlelNb4KRAAxAZclsJwwbbWoSEqyK4AZA== --osd-uuid 9863531a-af37-463c-95a1-920808222a67 2020-03-11T22:55:04.203 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:04.199+0000 7f57ceb9cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:04.205 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:04.202+0000 7f57ceb9cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:04.208 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:04.203+0000 7f57ceb9cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:04.208 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:04.204+0000 7f57ceb9cec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:55:04.208 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:04.204+0000 7f57ceb9cec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:55:04.208 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:04.204+0000 7f57ceb9cec0 -1 bluestore(td/osd-backfill-recovery-log/2/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/2/block: (2) No such file or directory 2020-03-11T22:55:04.330 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:04.327+0000 7f57ceb9cec0 -1 bluestore(td/osd-backfill-recovery-log/2) _read_fsid unparsable uuid 2020-03-11T22:55:07.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/2/keyring 2020-03-11T22:55:07.757 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:07.758 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd2 key to auth repository 2020-03-11T22:55:07.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd2 key to auth repository 2020-03-11T22:55:07.758 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:08.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.2 2020-03-11T22:55:08.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 2 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/2 --osd-journal=td/osd-backfill-recovery-log/2/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:08.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:08.211 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:08.212 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:08.212 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.2 2020-03-11T22:55:08.232 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:08.229+0000 7f2f9f30cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:08.234 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:08.231+0000 7f2f9f30cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:08.236 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:08.233+0000 7f2f9f30cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:08.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 2 2020-03-11T22:55:08.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:08.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=2 2020-03-11T22:55:08.558 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:08.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:08.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:08.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:08.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:08.559 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:55:08.559 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:55:08.751 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:08.748+0000 7f2f9f30cec0 -1 Falling back to public interface 2020-03-11T22:55:08.908 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:09.488 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:09.485+0000 7f2f9f30cec0 -1 osd.2 0 log_to_monitors {default=true} 2020-03-11T22:55:09.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:09.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:09.910 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:09.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:09.911 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.2 up' 2020-03-11T22:55:09.911 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:55:10.307 INFO:tasks.workunit.client.0.smithi155.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/85976,v1:127.0.0.1:6819/85976] [v2:127.0.0.1:6820/85976,v1:127.0.0.1:6821/85976] exists,up 9863531a-af37-463c-95a1-920808222a67 2020-03-11T22:55:10.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:55:10.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:55:10.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:55:10.307 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:10.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 3 2020-03-11T22:55:10.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:10.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:10.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=3 2020-03-11T22:55:10.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:10.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/3 2020-03-11T22:55:10.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:10.308 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:10.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:10.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:10.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/3' 2020-03-11T22:55:10.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/3/journal' 2020-03-11T22:55:10.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:10.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:55:10.309 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:10.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:10.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:10.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:10.310 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:10.310 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:10.310 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:55:10.310 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:55:10.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:55:10.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:10.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:10.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:10.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:10.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:10.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:10.311 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:10.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:10.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:10.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:10.312 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/3 2020-03-11T22:55:10.312 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:10.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=d9659571-a5a3-415f-b482-0207d3eda043 2020-03-11T22:55:10.313 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd3 d9659571-a5a3-415f-b482-0207d3eda043' 2020-03-11T22:55:10.313 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:10.313 INFO:tasks.workunit.client.0.smithi155.stdout:add osd3 d9659571-a5a3-415f-b482-0207d3eda043 2020-03-11T22:55:10.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBObGle9e73EhAA7vZItliosAtujstII+m16w== 2020-03-11T22:55:10.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBObGle9e73EhAA7vZItliosAtujstII+m16w=="}' 2020-03-11T22:55:10.321 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new d9659571-a5a3-415f-b482-0207d3eda043 -i td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:55:10.810 INFO:tasks.workunit.client.0.smithi155.stdout:3 2020-03-11T22:55:10.820 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/3/new.json 2020-03-11T22:55:10.821 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 3 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBObGle9e73EhAA7vZItliosAtujstII+m16w== --osd-uuid d9659571-a5a3-415f-b482-0207d3eda043 2020-03-11T22:55:10.844 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:10.841+0000 7f0befca0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:10.846 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:10.843+0000 7f0befca0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:10.848 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:10.845+0000 7f0befca0ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:10.849 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:10.846+0000 7f0befca0ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:55:10.849 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:10.846+0000 7f0befca0ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:55:10.849 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:10.846+0000 7f0befca0ec0 -1 bluestore(td/osd-backfill-recovery-log/3/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/3/block: (2) No such file or directory 2020-03-11T22:55:10.921 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:10.918+0000 7f0befca0ec0 -1 bluestore(td/osd-backfill-recovery-log/3) _read_fsid unparsable uuid 2020-03-11T22:55:14.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/3/keyring 2020-03-11T22:55:14.047 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:14.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd3 key to auth repository 2020-03-11T22:55:14.048 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:14.049 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd3 key to auth repository 2020-03-11T22:55:14.500 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.3 2020-03-11T22:55:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.3 2020-03-11T22:55:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 3 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/3 --osd-journal=td/osd-backfill-recovery-log/3/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:14.501 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:14.522 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:14.519+0000 7f9e32d0cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:14.523 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:14.521+0000 7f9e32d0cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:14.524 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:14.521+0000 7f9e32d0cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:14.857 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 3 2020-03-11T22:55:14.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:14.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=3 2020-03-11T22:55:14.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:14.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:14.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:14.858 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:14.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:55:14.859 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:14.859 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:55:15.041 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:15.037+0000 7f9e32d0cec0 -1 Falling back to public interface 2020-03-11T22:55:15.207 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:15.778 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:15.776+0000 7f9e32d0cec0 -1 osd.3 0 log_to_monitors {default=true} 2020-03-11T22:55:16.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:16.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:16.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:16.208 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:16.209 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.3 up' 2020-03-11T22:55:16.209 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:55:16.564 INFO:tasks.workunit.client.0.smithi155.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/86331,v1:127.0.0.1:6827/86331] [v2:127.0.0.1:6828/86331,v1:127.0.0.1:6829/86331] exists,up d9659571-a5a3-415f-b482-0207d3eda043 2020-03-11T22:55:16.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:55:16.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:55:16.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:55:16.564 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:16.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 4 2020-03-11T22:55:16.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:16.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:16.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=4 2020-03-11T22:55:16.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:16.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/4 2020-03-11T22:55:16.565 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:16.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:16.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:16.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:16.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/4' 2020-03-11T22:55:16.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/4/journal' 2020-03-11T22:55:16.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:16.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:55:16.566 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:16.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:16.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:16.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:16.567 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:16.567 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:16.567 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:55:16.567 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:55:16.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:55:16.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:16.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:16.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:16.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:16.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:16.568 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:16.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:16.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:16.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:16.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:16.569 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/4 2020-03-11T22:55:16.569 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:16.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=531b5db2-3dbb-469e-aa24-d14a9d953fc5 2020-03-11T22:55:16.570 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd4 531b5db2-3dbb-469e-aa24-d14a9d953fc5' 2020-03-11T22:55:16.570 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:16.570 INFO:tasks.workunit.client.0.smithi155.stdout:add osd4 531b5db2-3dbb-469e-aa24-d14a9d953fc5 2020-03-11T22:55:16.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBUbGle77ZHIhAANf8T48TmKkPMee45bYINyw== 2020-03-11T22:55:16.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBUbGle77ZHIhAANf8T48TmKkPMee45bYINyw=="}' 2020-03-11T22:55:16.578 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 531b5db2-3dbb-469e-aa24-d14a9d953fc5 -i td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:55:17.042 INFO:tasks.workunit.client.0.smithi155.stdout:4 2020-03-11T22:55:17.052 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/4/new.json 2020-03-11T22:55:17.053 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 4 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBUbGle77ZHIhAANf8T48TmKkPMee45bYINyw== --osd-uuid 531b5db2-3dbb-469e-aa24-d14a9d953fc5 2020-03-11T22:55:17.077 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:17.074+0000 7f7b076d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:17.078 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:17.076+0000 7f7b076d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:17.079 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:17.077+0000 7f7b076d4ec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:17.080 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:17.077+0000 7f7b076d4ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:55:17.080 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:17.077+0000 7f7b076d4ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:55:17.080 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:17.077+0000 7f7b076d4ec0 -1 bluestore(td/osd-backfill-recovery-log/4/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/4/block: (2) No such file or directory 2020-03-11T22:55:17.186 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:17.183+0000 7f7b076d4ec0 -1 bluestore(td/osd-backfill-recovery-log/4) _read_fsid unparsable uuid 2020-03-11T22:55:20.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: local key_fn=td/osd-backfill-recovery-log/4/keyring 2020-03-11T22:55:20.583 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: cat 2020-03-11T22:55:20.584 INFO:tasks.workunit.client.0.smithi155.stdout:adding osd4 key to auth repository 2020-03-11T22:55:20.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: echo adding osd4 key to auth repository 2020-03-11T22:55:20.584 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:673: run_osd: ceph -i td/osd-backfill-recovery-log/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2020-03-11T22:55:21.042 INFO:tasks.workunit.client.0.smithi155.stdout:start osd.4 2020-03-11T22:55:21.042 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:674: run_osd: echo start osd.4 2020-03-11T22:55:21.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: ceph osd dump --format=json 2020-03-11T22:55:21.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: ceph-osd -i 4 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/4 --osd-journal=td/osd-backfill-recovery-log/4/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:21.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: jq '.flags_set[]' 2020-03-11T22:55:21.043 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:678: run_osd: grep -q '"noup"' 2020-03-11T22:55:21.064 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:21.061+0000 7ff67ab7cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:21.065 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:21.063+0000 7ff67ab7cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:21.066 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:21.064+0000 7ff67ab7cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:21.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:681: run_osd: wait_for_osd up 4 2020-03-11T22:55:21.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:915: wait_for_osd: local state=up 2020-03-11T22:55:21.389 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: wait_for_osd: local id=4 2020-03-11T22:55:21.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: wait_for_osd: status=1 2020-03-11T22:55:21.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i=0 )) 2020-03-11T22:55:21.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:21.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 0 2020-03-11T22:55:21.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:21.390 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:55:21.390 INFO:tasks.workunit.client.0.smithi155.stdout:0 2020-03-11T22:55:21.584 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:21.582+0000 7ff67ab7cec0 -1 Falling back to public interface 2020-03-11T22:55:21.738 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:922: wait_for_osd: sleep 1 2020-03-11T22:55:22.338 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:22.335+0000 7ff67ab7cec0 -1 osd.4 0 log_to_monitors {default=true} 2020-03-11T22:55:22.739 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i++ )) 2020-03-11T22:55:22.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: wait_for_osd: (( i < 300 )) 2020-03-11T22:55:22.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:920: wait_for_osd: echo 1 2020-03-11T22:55:22.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: ceph osd dump 2020-03-11T22:55:22.740 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: wait_for_osd: grep 'osd.4 up' 2020-03-11T22:55:22.740 INFO:tasks.workunit.client.0.smithi155.stdout:1 2020-03-11T22:55:23.164 INFO:tasks.workunit.client.0.smithi155.stdout:osd.4 up in weight 1 up_from 23 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/86694,v1:127.0.0.1:6835/86694] [v2:127.0.0.1:6836/86694,v1:127.0.0.1:6837/86694] exists,up 531b5db2-3dbb-469e-aa24-d14a9d953fc5 2020-03-11T22:55:23.164 INFO:tasks.workunit.client.0.smithi155.stdout:add osd5 23aa038a-65cd-4af9-9102-3ec8a5e3ba68 2020-03-11T22:55:23.164 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:924: wait_for_osd: status=0 2020-03-11T22:55:23.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: wait_for_osd: break 2020-03-11T22:55:23.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:928: wait_for_osd: return 0 2020-03-11T22:55:23.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:54: _common_test: for osd in $(seq 0 $(expr $OSDS - 1)) 2020-03-11T22:55:23.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-backfill-recovery-log.sh:56: _common_test: run_osd td/osd-backfill-recovery-log 5 2020-03-11T22:55:23.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: local dir=td/osd-backfill-recovery-log 2020-03-11T22:55:23.165 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: shift 2020-03-11T22:55:23.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: local id=5 2020-03-11T22:55:23.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: shift 2020-03-11T22:55:23.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: local osd_data=td/osd-backfill-recovery-log/5 2020-03-11T22:55:23.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: local 'ceph_args=--fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 ' 2020-03-11T22:55:23.166 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2020-03-11T22:55:23.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --osd-journal-size=100' 2020-03-11T22:55:23.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2020-03-11T22:55:23.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --osd-data=td/osd-backfill-recovery-log/5' 2020-03-11T22:55:23.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --osd-journal=td/osd-backfill-recovery-log/5/journal' 2020-03-11T22:55:23.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --chdir=' 2020-03-11T22:55:23.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2' 2020-03-11T22:55:23.167 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --run-dir=td/osd-backfill-recovery-log' 2020-03-11T22:55:23.167 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: get_asok_path 2020-03-11T22:55:23.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:116: get_asok_path: local name= 2020-03-11T22:55:23.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:117: get_asok_path: '[' -n '' ']' 2020-03-11T22:55:23.168 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: get_asok_dir 2020-03-11T22:55:23.168 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_dir: '[' -n '' ']' 2020-03-11T22:55:23.168 INFO:tasks.workunit.client.0.smithi155.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_dir: echo /tmp/ceph-asok.76480 2020-03-11T22:55:23.168 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:120: get_asok_path: echo '/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:55:23.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' 2020-03-11T22:55:23.168 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --debug-osd=20' 2020-03-11T22:55:23.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --debug-ms=1' 2020-03-11T22:55:23.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' --debug-monc=20' 2020-03-11T22:55:23.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=' --log-file=td/osd-backfill-recovery-log/$name.log' 2020-03-11T22:55:23.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: ceph_args+=' --pid-file=td/osd-backfill-recovery-log/$name.pid' 2020-03-11T22:55:23.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:652: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2020-03-11T22:55:23.169 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2020-03-11T22:55:23.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2020-03-11T22:55:23.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph_args+=' ' 2020-03-11T22:55:23.170 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: ceph_args+= 2020-03-11T22:55:23.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: mkdir -p td/osd-backfill-recovery-log/5 2020-03-11T22:55:23.171 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: uuidgen 2020-03-11T22:55:23.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: local uuid=23aa038a-65cd-4af9-9102-3ec8a5e3ba68 2020-03-11T22:55:23.171 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:660: run_osd: echo 'add osd5 23aa038a-65cd-4af9-9102-3ec8a5e3ba68' 2020-03-11T22:55:23.171 INFO:tasks.workunit.client.0.smithi155.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: ceph-authtool --gen-print-key 2020-03-11T22:55:23.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: OSD_SECRET=AQBbbGle8LxuChAAgsAWXO2mSiuFyfDwUurV6g== 2020-03-11T22:55:23.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: echo '{"cephx_secret": "AQBbbGle8LxuChAAgsAWXO2mSiuFyfDwUurV6g=="}' 2020-03-11T22:55:23.178 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:663: run_osd: ceph osd new 23aa038a-65cd-4af9-9102-3ec8a5e3ba68 -i td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:55:23.742 INFO:tasks.workunit.client.0.smithi155.stdout:5 2020-03-11T22:55:23.751 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:664: run_osd: rm td/osd-backfill-recovery-log/5/new.json 2020-03-11T22:55:23.752 INFO:tasks.workunit.client.0.smithi155.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:665: run_osd: ceph-osd -i 5 --fsid=c21024d3-ba22-4e3f-8c59-f8d1c3bbe978 --auth-supported=none --mon-host=127.0.0.1:7129 --osd_max_backfills=1 --debug_reserver=20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/osd-backfill-recovery-log/5 --osd-journal=td/osd-backfill-recovery-log/5/journal --chdir= --osd_min_pg_log_entries=1 --osd_max_pg_log_entries=2 --run-dir=td/osd-backfill-recovery-log '--admin-socket=/tmp/ceph-asok.76480/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/osd-backfill-recovery-log/$name.log' '--pid-file=td/osd-backfill-recovery-log/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --mkfs --key AQBbbGle8LxuChAAgsAWXO2mSiuFyfDwUurV6g== --osd-uuid 23aa038a-65cd-4af9-9102-3ec8a5e3ba68 2020-03-11T22:55:23.775 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:23.773+0000 7fb7a194cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:23.777 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:23.774+0000 7fb7a194cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:23.780 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:23.776+0000 7fb7a194cec0 -1 WARNING: all dangerous and experimental features are enabled. 2020-03-11T22:55:23.781 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:23.776+0000 7fb7a194cec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:55:23.781 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:23.776+0000 7fb7a194cec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:55:23.781 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:23.776+0000 7fb7a194cec0 -1 bluestore(td/osd-backfill-recovery-log/5/block) _read_bdev_label failed to open td/osd-backfill-recovery-log/5/block: (2) No such file or directory 2020-03-11T22:55:23.819 INFO:tasks.workunit.client.0.smithi155.stderr:2020-03-11T22:55:23.816+0000 7fb7a194cec0 -1 bluestore(td/osd-backfill-recovery-log/5) _read_fsid unparsable uuid 2020-03-11T22:55:24.939 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2020-03-11T22:55:24.939 DEBUG:teuthology.task.console_log:Killing console logger for smithi155 2020-03-11T22:55:24.940 DEBUG:teuthology.task.console_log:Killing console logger for smithi155 2020-03-11T22:55:24.940 DEBUG:teuthology.exit:Finished running handlers